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ABSTRACT 


The  development  of  the  Logistics  Management  Decision  Support  System 
(LMDSS)  by  the  Naval  Air  Systems  Command  (NAVAIR)  has  been  on  going 
since  1991.  Originally  conceived  as  a  strategic  Decision  Support  System  (DSS), 
LMDSS  has  instead  evolved  into  a  Web  portal  for  data  analysts  to  access 
NAVAIR's  Aviation  Maintenance  and  Material  Management  (AV-3M)  data. 
LMDSS  is  more  accurately  described  as  a  Web-based,  Management  Information 
System  (MIS)  than  as  a  DSS. 

This  research  examines  the  information  needs  and  decision  process 
requirements  of  LMDSS  users.  Focus  groups,  interviews,  and  a  Web-based 
survey  were  conducted  to  collect  decision  support  and  data  requirements  from 
Fleet  customers.  User  perceptions,  feedback,  and  recommendations  to  improve 
LMDSS  are  described  and  analyzed.  Historical  insights  into  the  development 
histoiy  of  LMDSS  are  introduced  as  a  lessons  learned  for  future  NAVAIR 
software  teams. 

Problems  identified  by  users  are  presented  followed  by  specific 
recommendations  for  solutions.  A  protype  model  developed  for  this  thesis  is 
found  in  the  appendices  as  an  example  of  how  a  modeling  capability  could 
enhance  the  ability  of  LMDSS  to  better  support  strategic,  unstructured  decisions.. 
Recommendations  are  provided  for  future  research. 
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L  INTRODUCTION 


A.  BACKGROUND 

The  first  effort  toward  building  a  Decision  Support  System  (DSS)  for  program 
managers  at  the  Naval  Air  Systems  Command  (NAVAIR)  began  in  1991.  NAVAIR's 
Program  Managers,  Air  (PMAs)  and  Assistant  Program  Managers,  Logistics  (APMLs) 
needed  a  tool  to  help  them  investigate  alternatives  and  make  optimal,  unstructured 
decisions  in  their  efforts  to  reduce  life  cycle  program  costs  while  maintaining  readiness. 
In  1994,  such  a  tool  was  completed  and  named  the  Logistics  Management  Decision 
Support  System  (LMDSS). 

When  Rear  Admiral  Donald  Eaton  (NAVAIR  4.0)  and  others  were  briefed  on  the 
LMDSS  program  in  1994,  many  were  excited  by  its  potential.  One  APML  even  reported 
attaining  a  $42  million  cost  avoidance  when  LMDSS  was  used  to  identify  and  solve  a 
problem  in  the  F-14  avionics  system.  It  appeared  that  LMDSS  might  just  be  the  tool 
PMAs  needed  to  better  manage  the  life  cycle  support  costs  of  their  programs  as  harsh 
budget  cuts  loomed  on  the  horizon. 

Years  later,  as  the  Logistics  Program  Department  Head  at  the  Naval  Postgraduate 
School,  Rear  Admiral  (retired)  Donald  Eaton  wondered  whatever  became  of  the  LMDSS 
program  that  showed  so  much  promise  back  in  1994.  One  of  the  goals  of  this  thesis  is  to 
find  out  what  happened  to  the  LMDSS  program.  Does  LMDSS  currently  meet  the 
requirements  of  its  users? 

B.  OBJECTIVES 

The  primary  objective  of  this  research  is  to  investigate  the  information  needs  and 
,  decision  process  requirements  of  current  LMDSS  users.  A  secondary  objective  is  to 
document  how  and  why  LMDSS  took  eight  years  to  transition  from  a  strategic  level  DSS 
to  a  Web  based  Relational  Database  Management  System  (RDBMS)  interface  for  Fleet 
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data  analysts  at  all  levels.  A  final  goal  is  to  provide  our  sponsor,  NAVAIR  3.6.2,  with 
user  feedback  and  research  results  to  assist  in  the  management  of  the  LMDSS  program. 

C.  RESEARCH  QUESTIONS 

The  primary  research  question  is:  "Does  LMDSS  meet  user  requirements?"  Other 
major  research  questions  are: 

•  How  can  LMDSS  be  improved  to  better  meet  managerial  needs? 

•  What  are  the  significant  decisions  users  would  like  LMDSS  to  support? 

•  How  has  LMDSS  been  funded? 

•  How  can  NAVAIR  improve  the  LMDSS  program? 

•  What  can  future  software  developers  learn  from  studying  the  LMDSS  project? 

•  What  are  the  key  attitudes  and  opinions  of  LMDSS  users? 

D.  ORGANIZATION  OF  THE  THESIS 

This  study  documents  some  of  the  key  events  that  occurred  during  the  eight-year 
effort  to  develop  LMDSS  and  examines  whether  or  not  LMDSS  currently  meets  the 
requirements  of  its  users.  Chapter  II  is  a  review  of  previous  LMDSS  research,  DSS 
literature,  and  results  from  current  software  development  research.  Chapter  HI  reviews 
the  history  of  the  LMDSS  project  and  discusses  how  it  has  been  funded.  Chapter  IV 
documents  a  detailed  description  of  the  research  methodology  used  in  this  thesis.  Chapter 
V  reports  the  research  findings.  Chapter  VI  presents  an  analysis  of  the  findings  and 
provides  answers  to  the  above  research  questions.  Chapter  VII  provides  a  continued 
discussion  of  what  the  research  results  mean  with  conclusions  and  corresponding 
recommendations.  A  prototype  model  is  presented  in  the  appendix  section  as  an  example 
to  the  reader  of  how  modeling  can  assist  managers  in  making  strategic,  unstructured 
decisions. 
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IL  LITERATURE  REVIEW 


A.  PREVIOUS  LMDSS  RESEARCH 

1.  LMDSS  Overview 

The  LMDSS  concept  was  first  introduced  by  the  Naval  Air  Systems  Command 
(NAVAIR)  and  the  Naval  Inventory  Control  Point  (NAVICP),  formally  called  the 
Aviation  Supply  Office  (ASO),  in  1991.  The  concept  involved  the  development  of  a 
decision  support  system  (DSS)  to  assist  NAVAIR  logistics  and  weapon  system  managers 
in  reducing  the  life  cycle  support  costs  of  aviation  systems  while  maintaining  readiness. 
Although  the  LMDSS  application  was  completed,  delivered,  and  operational  for  a  short 
period  in  1994,  it  was  placed  back  into  development  to  be  restructured  as  a  World  Wide 
Web  (WWW)  based  tool.  As  of  the  summer  of  1999,  LMDSS  is  still  considered  a 
prototype  rather  than  a  fully  functional  software  application. 

The  only  published  research  that  specifically  addresses  the  LMDSS  application 
was  completed  by  Moore  and  Snyder  (1998)  at  the  Naval  Postgraduate  School.  They 
assessed  the  capability  of  LMDSS  to  meet  the  information  requirements  of  NAVAIR's 
logistics  managers.  They  also  gauged  whether  the  LMDSS  prototype's  capabilities 
satisfied  the  criteria  of  a  DSS. 

LMDSS  is  unusual  because  of  the  time  it  has  been  under  development  (over  8 
years).  As  a  result,  we  chose  to  expand  the  focus  of  research  to  include  software 
development  and  acquisition  issues  in  addition  to  user  requirements.  The  goal  is  to  help 
explain  how  LMDSS  grew  from  a  three-year  project  to  a  seemingly  endless  software 
development  effort. 

2.  LMDSS  Research  Findings 

Moore  and  Snyder  (1998)  concluded  that: 

•  Because  of  its  lack  of  modeling  and  sensitivity  analysis,  LMDSS  is  not  a 
DSS.  It  is  actually  a  relational  database  management  system  (RDBMS) 
interface  that  improves  data  accessibility. 
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•  APMLs  are  only  one  of  many  potential  user  groups  of  LMDSS. 

•  LMDSS  meets  the  requirements  of  surveyed  logistics  managers  to  support 
Affordable  Readiness  initiatives. 

•  Lack  of  modeling,  graphics,  and  sensitivity  analysis  capabilities  limits  the 
identification,  analysis,  and  comparison  of  Affordable  Readiness 
initiatives  using  LMDSS. 

•  Data  quality  is  both  a  real  and  perceived  problem. 

•  The  environment  in  which  aviation  program  management  decisions  are 
made  limits  the  effectiveness  of  LMDSS  in  measurably  reducing  life-cycle 
costs. 

a)  Research  Weaknesses 

Moore  and  Snyder's  (1998)  research  targeted  NAVAIR's  APMLs  as  the 
primary  users  of  LMDSS.  Survey  results  from  the  nine  APMLs  contacted  revealed  only 
one  respondent  had  personally  used  LMDSS.  Their  interviews  revealed  each  of  the 
APML  program  offices  had  its  own  unique  strategy  for  accomplishing  logistics  and 
maintenance  data  analysis  and  collection.  Some  offices  hired  civilian  contractors  while 
others  used  various  mixtures  of  in-house  personnel  and  personnel  assigned  to  lower 
echelon  commands.  They  discovered  data  analysts  rather  than  APMLs  and  PMAs  were 
the  actual  users  of  LMDSS.  Although  LMDSS  was  originally  designed  to  support  PMAs 
and  APMLs,  its  use  has  been  delegated  to  lower  level  analysts.  As  a  result,  the  LMDSS 
prototype  has  evolved  from  a  PMA/APML  decision  tool  to  a  RDBMS  interface  for  Fleet 
data  analysts.  The  requirements  identified  by  the  nine  APMLs  in  Moore  and  Snyder’s 
thesis  are  not  representative  of  the  needs  of  the  larger,  more  varied  population  of  over 
800  LMDSS  users. 

The  fact  that  LMDSS  is  still  in  the  prototype  phase  and  thus  not  widely 
'  used  was  mentioned  as  a  significant  research  constraint  (Moore  and  Snyder,  1998). 
Although  LMDSS  was  scheduled  to  be  operational  by  the  summer  of  1998,  it  was  still 
undergoing  quality  assurance  testing  as  of  the  summer  of  1999.  The  constraints  posed 


4 


when  querying  users  about  a  prototype  software  tool  that  was  not  fully  operational 
continue  to  be  a  problem  for  researchers. 

b)  Research  Strengths 

Moore  and  Snyder's  (1998)  thesis  work  reiterated  how  important  it  is  for 
NAV AIR's  PMAs  to  make  smart  decisions  concerning  life  cycle  support  costs.  The  data 
collected  showed  that  PMAs  require  more  than  a  measuring  tool  to  display  costs.  To 
make  optimal  decisions,  PMAs  and  APMLs  need  the  ability  to  test  alternative  courses  of 
action  to  see  how  controllable  and  uncontrollable  variables  affect  life-cycle  costs  and 
readiness.  According  to  the  APMLs,  a  fully  functional  DSS  is  still  needed  at  the  PMA 
and  APML  levels  to  make  such  decisions. 

Previous  research  verified  that  LMDSS  could  not  be  classified  as  a  DSS 
due  to  its  inability  to  support  modeling,  graphics,  and  sensitivity  analysis.  Such 
limitations  significantly  degrade  LMDSS's  usefulness  as  an  effective  decision  support 
tool  for  the  PMAs  and  APMLs.  One  APML  that  Moore  and  Snyder  interviewed  stated, 

LMDSS  is  good  at  "big  picture"  and  indicating  "where  to  go  look" 
but  falls  short  of  communicating  details  with  ease  or  indicating  "why"  a 
system  measurement  is  as  it  is... "LMDSS  can  help  answer  the  what,  but  it  ' 
can't  help  me  with  the  why  or  the  how."  (Moore  and  Snyder,  1998,  p.  86) 

Moore  and  Snyder  (1998)  identified  problem  areas  in  the  LMDSS 
environment  that  significantly  degraded  its  effectiveness.  Data  quality  issues,  difficulties 
with  depot  cost  data,  and  the  need  for  users  to  understand  the  origin  and  derivation  of 
data  to  fully  utilize  and  trust  LMDSS  were  noteworthy  findings.  It  was  noted  that  the 
environment  of  a  typical  PMA  did  not  encourage  or  reward  risk  taking  and  creativity 
when  considering  life  cycle  costs.  They  wrote. 

The  current  environment  encourages  short-term  decisions  that 
compromise  life-cycle  decisions.  The  LMDSS  can  help  identify  and 
justify  decisions  to  reduce  life-cycle  costs,  but  other  factors  are  driving  up 
these  same  costs.  (Moore  and  Snyder,  1998,  p.  93) 
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Instances  where  NAVAIR  failed  to  communicate  adequately  with  LMDSS 
users  were  also  mentioned  as  potential  problems.  An  example  was  the  LMDSS  web  site's 
failure  to  clearly  identify  itself  as  a  prototype  site  with  only  a  partial  database.  This 
caused  numerous  misunderstandings  at  the  user  level.  Such  communication  errors 
directly  affected  user  perceptions  about  LMDSS's  relevancy  and  led  to  a  lack  of 
acceptance  and  trust.  (Moore  and  Snyder,  1998) 

3.  Identification  of  LMDSS  Users 

Moore  and  Snyder  (1998)  identified  only  one  APML  who  was  an  actual  LMDSS 
user.  Most  of  the  real  users  were  found  to  be  military  and  civilian  data  analysts  at  all 
levels  within  the  Fleet.  Much  of  the  earlier  DSS  literature  suggested  the  use  of  DSS 
might  be  more  appropriate  at  only  the  higher  levels  of  management.  This  was  the  view  of 
those  who  originally  developed  LMDSS  back  in  1991.  More  recent  DSS  literature  now 
claims  the  use  of  a  DSS  is  appropriate  at  any  management  level.  (Ariav  and  Ginsberg, 
1985) 

While  much  of  the  literature  emphasizes  DSS  use  by  top  management,  in  reality, 
middle  management  and  professional  staffers  are  often  the  real  hands-on  users  of  a  DSS 
(Sprague,  1996). 

This  does  not  mean  that  DSS  is  not  used  by  top  management.  Rather, 
what  often  happens  is  that  senior  executives  request  information  from  a 
staff  assistant  who  uses  a  DSS  to  obtain  information.  (Sprague,  1996,  p. 

103) 

This  finding  appears  to  confirm  what  Moore  and  Snyder  uncovered  in  their  thesis. 
The  PMAs  and  APMLs  contacted  by  them  admitted  delegating  the  use  of  the  LMDSS 
application  to  lower  level  analysts.  This  delegation  provides  some  insight  into  how 
LMDSS  transitioned  from  a  strategic  level  decision  tool  to  an  analysis  tool  for  Fleet  data 
analysts. 

One  study  conducted  by  Snead  and  Harrell  discussed  the  gap  between  the 
capabilities  of  sophisticated  computer-based  information  systems  (CBIS)  and  the  extent 
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they  are  used  by  managers.  Their  conclusions  emphasized  the  important  role  a  manager's 
expectations  play  in  whether  or  not  he  accepts  and  uses  a  CBIS.  A  strategy  that  includes 
more  extensive  manager  involvement  and  training  to  support  the  use  of  a  CBIS  was 
strongly  recommended.  The  study  suggests  more  aggressive  training  and  close  user 
involvement  may  be  the  key  to  getting  NAVAIR's  logistics  and  maintenance  managers  to 
accept  and  use  LMDSS  when  it  becomes  fully  operational.  (Snead  and  Harrell,  1994) 

A  typical  DSS  user  has  much  more  latitude  to  ignore  or  circumvent  the  system 
than  users  of  other  more  traditional  CBIS.  Research  emphasizes  how  important  it  is  for  a 
DSS  to  earn  the  allegiance  of  its  users  by  being  valuable  and  convenient.  (Sprague,  1980) 
Little  (1970)  called  for  DSSs  to  be  simple,  robust,  easy  to  control  and  adaptive  in  order 
gain  management  support. 

B.  DECISION  SUPPORT  SYSTEM  LITERATURE 

On  NAVAIR's  web  site,  the  LMDSS  application  is  presented  as  a  DSS  which 
integrates  data  from  maintenance,  flight,  cost,  and  material  data  bases  into  a  structured 
and  repeatable  decision  making  process.  In  numerous  NAVAlR  briefs  and  plans, 
LMDSS  is  shown  as  Naval  Aviation  Logistics  Data  and  Analysis  (NALDA)  ITs  primary 
analysis  tool  for  historical  Aviation  Maintenance  and  Material  Management  (AV3M) 
data  within  NAVAIR's  Integrated  Weapons  Systems  Database  (IWSDB).  (NAVAIR, 
1999) 

To  avoid  misunderstandings  when  discussing  a  DSS,  it  is  important  to  review  the 
basic  DSS  definitions  and  terms  provided  by  the  literature. 

1.  What  is  a  DSS? 

An  accepted  definition  of  a  DSS  is  one  provided  by  Bhargava,  Krishman,  and 
Whinston  (1994,  p.  2): 
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Traditional  decision  support  systems  (DSS)  support  individuals 
making  semi-structured  decisions  in  which  mathematical  (or  other  formal) 
models  are  used  for  the  structured  parts,  leaving  the  decision  maker  to 
exercise  judgement  in  handling  the  unstructured  parts.  Focusing  on  the 
choice-related  tasks,  DSS  facilitate  the  use  of  formal  modeling  techniques 
in  making  complex  decisions.  Among  the  benefits  claimed  for  these 
systems  is  that  they  facilitate  the  investigation  of  more  alternatives,  and 
support  ad  hoc  query  and  analysis. 


Another  definition  defines  a  DSS  as  a  computer-based  information  system,  which 
provides  interactive  information  support  to  managers  making  semi-structured  and 
unstructured  decisions.  A  DSS  supports  managers  by  using,  (1)  analytical  models,  (2) 
specialized  databases,  (3)  a  decision-maker’s  own  insights  and  judgements,  and  (4)  an 
interactive,  computer-based  modeling  process.  (O'Brian,  1993) 

Some  terms  in  the  above  DSS  definitions  need  to  be  clarified.  An  ad  hoc  query  is 
a  unique,  situation-specific,  unscheduled  request  for  information.  A  structured  decision 
occurs  in  situations  where  the  procedures  to  follow  in  making  a  decision  can  be  predicted 
and  specified  in  advance.  Often,  the  outcome  of  structured  decisions  can  be  predicted 
with  relative  certainty.  An  unstructured  decision  involves  situations  where  it  is  not 
possible  to  know  in  advance  what  procedures  to  follow.  Unstructured  decisions  involve 
too  many  random  events,  unknown  variables,  and  hidden  relationships  to  be  able  to 
follow  any  preset  rules  or  checklists.  A  semi-structured  decision  is  one  where  some  of 
the  decision  procedures  can  be  pre-specified,  but  not  to  an  extent  where  it  can  lead  to  a 
definite  decision.  (OBrian,  1993) 

Currently  the  LMDSS  application  can  support  structured  decisions  and  ad  hoc 
queries,  but  is  unable  to  assist  with  unstructured  or  semi-structured  decisions  due  to  the 
absence  of  modeling  and  sensitivity  analysis. 

2.  What  is  the  Decision-Making  Process? 

A  classic  model  of  a  manager's  decision  process  is  depicted  by  Herbert  Simon,  a 
leader  in  the  field  of  decision  theory,  who  presented  decision-making  as  consisting  of 
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three  phases:  intelligence,  design,  and  choice.  The  intelligence  phase  involves  detecting 
problems,  collecting  problem  data,  and  analysis  of  the  environment.  The  design  phase 
includes  setting  criteria  and  objectives,  creating  alternatives,  and  evaluating  the  outcome. 
The  choice  phase  consists  of  determining  the  outcome  of  selected  alternatives  and 
selecting  the  desired  outcome  consistent  with  the  decision  objectives.  Simon's  work  is 
important  because  it  built  the  theoretical  foundation  still  used  by  software  engineers  to 
build  DSSs.  (Simon,  1977) 

Steven  Alter  added  a  fourth  phase  to  Simon's  model  calling  it  the  implementation 
phase.  This  phase  involved  the  process  of  putting  a  manager's  decision  into  effect.  It 
included  building  a  consensus  to  support  the  decision,  explaining  the  decision  to  the  right 
people,  and  fostering  the  commitment  to  follow  through  with  the  decision.  Alter's 
research  into  this  implementation  phase  revealed  a  DSS's  greatest  impact  was  not  in 
aiding  the  decision-making  process  but  with  explaining  and  justifying  decisions  already 
made.  (Alter,  1980) 

3.  Why  do  PMAs  and  APMLs  need  a  DSS? 

Most  researchers  agree  that  managers  are  not  completely  rational  decision¬ 
makers.  An  ideal  rational  decision-maker  performs  all  three  phases  of  Simon's  model 
flawlessly.  Such  a  decision-maker  ensures  all  information  is  gathered  and  interpreted  in 
an  unbiased  maimer.  He  identifies  all  practical  alternatives  and  evaluates  them  using 
unambiguous  criteria.  He  then  selects  the  best  alternative  based  on  a  consistent  and 
explicit  set  of  weightings  and  trade-offs.  (Alter,  1992) 

In  the  real  world,  however,  managers  don't  have  time  to  collect  all  decision- 
related  information  and  fully  explore  all  feasible  alternatives.  Clear,  focused  objectives 
and  criteria  are  often  not  completely  defined  with  more  weight  placed  on  intangible 
information.  Alter  (1992)  identified  eight  common  managerial  decision-making  flaws: 

•  Poor  Framing:  Allowing  a  decision  to  be  influenced  excessively  by  the 
language  used  in  describing  the  decision 

•  Recency  Effects:  Giving  undue  weight  to  the  most  recent  information 
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•  Primacy  Effects:  Giving  undue  weight  to  the  first  information  received 

•  Poor  Probability  Estimation:  Overestimating  the  probability  of  familiar  or 
dramatic  events;  underestimating  the  probably  of  negative  events 

•  Overconfidence:  Believing  too  strongly  in  one's  own  knowledge 

•  Escalation  Phenomena:  Unwillingness  to  abandon  courses  of  action  that 
have  been  decided  upon  previously 

•  Association  Bias:  Reusing  strategies  that  were  successful  in  the  past, 
regardless  of  whether  they  fit  the  current  situation 

•  Groupthink:  Overemphasizing  group  consensus  and  cohesiveness  instead 
of  bringing  out  unpopular  ideas 

Simon  developed  the  idea  that  managers  "satisfice"  by  choosing  a  satisfactory 
course  of  action  that  is  "good  enough".  His  research  found  human  decision  makers  have 
limited  information  processing  capabilities  and  resort  to  satisficing  when  faced  with  time 
constraints,  minimal  information,  and  a  limited  ability  to  process  all  relevant 
information.  (Simon,  1977) 

The  purpose  of  a  DSS  is  to  convert  decision  making  from  an  art  into  a  scientific, 
more  rational  approach.  Relying  on  a  DSS  can  help  managers  avoid  some  of  the  common 
mistakes  listed  above.  A  DSS  can  be  also  be  an  invaluable  tool  to  justify  NAVAIR 
programs  and  decisions  made  by  PMAs  and  APMLs.  A  DSS  can  help  a  manager  frame  a 
problem,  collect  more  pertinent  information,  develop  and  evaluate  feasible  courses  of 
action,  test  assumptions,  and  select  the  best  solution  to  achieve  the  desired  objectives. 
The  original  vision  of  the  LMDSS  application  was  to  provide  such  support  to  PMAs  and 
APMLs  in  order  to  achieve  life  cycle  cost  savings  and  improve  readiness  in  an 
environment  of  shrinking  budget  authority. 

The  literature  concerning  DSS  architecture  contains  various  structural  views  of  a 
DSS  concept  with  each  author  using  unique  nomenclature  to  describe  the  parts.  Most 
descriptions  break  down  a  DSS  into  three  conceptual  components:  (1)  a  user/system 
interface,  (2)  data  management,  (5)  and  model  management  (Ariav  and  Ginzberg,  1985). 
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One  view  of  a  DSS  describes  it  using  a  Dialog  Data  Model  (DDM)  paradigm. 
This  paradigm  states  all  DSSs  contain,  in  one  form  or  another,  three  major  components:  a 
Dialog  between  the  user  and  the  system,  the  Data  supporting  the  DSS,  and  Models  that 
provide  the  analysis  (Sprague,  1996).  Our  search  of  the  DSS  literature  was  unable  to 
uncover  any  author  who  did  not  list  a  model  base  or  model  capability  as  a  key  component 
of  a  DSS. 

4.  Modeling  within  Decision  Support  Systems 

a)  Capabilities  of  a  DSS  Model 

Models  provide  powerful  analytical  capabilities  to  a  DSS.  The  capability 
of  a  DSS  to  simulate  a  process  with  a  model  and  conduct  what-if  analysis  is  what 
separates  it  from  a  traditional  CBIS.  Sprague  (1980),  Ariav  and  Ginzberg  (1985) 
determined  the  key  capabilities  of  the  model  component  within  a  DSS  to  include: 

•  The  ability  to  easily  create  new  models  and  update  parameters 

•  Support  of  a  model  directory  containing  all  available  models 

•  The  capability  to  interface  with  databases  to  retrieve  data,  store  model 

output,  and  input  data  to  other  models  through  databases 

•  Access  to  a  library  of  model  "building  blocks" 

•  Support  of  a  Model  Base  Management  System  (MBMS)  containing 

mechanisms  to  store,  catalog,  link  and  access  models 

b)  Four  Types  of  Modeling  Activities 

The  four  types  of  analytical  modeling  that  can  be  found  in  a  DSS  are:  (1) 
what-if  analysis  (2)  sensitivity  analysis  (3)  goal  seeking  analysis  and  (4)  optimization 
analysis  (O'Brian,  1993). 

What-if  analysis  involves  making  changes  to  process  variables  or  variable 
relationships  and  observing  the  resulting  changes  in  the  output  variables.  Sensitivity 
analysis  is  a  special  case  where  only  one  variable  is  changed  repeatedly  to  see  the  affect 
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on  other  system  variables.  Goal  seeking  analysis  sets  a  target  value  or  goal  and  changes 
various  variables  until  the  goal  is  achieved  Optimization  analysis  is  finding  the  optimum 
value  for  selected  variables,  given  certain  constraints.  (O'Brian,  1993) 

c)  Why  Models  are  Hard  to  Implement 

The  difficulties  inherent  in  building  models  today  are  the  very  same  ones 
Little  (1970)  wrote  of  in  his  often  cited  research.  His  conclusions  may  help  to  explain 
why  LMDSS  does  not  currently  contain  models.  He  noted  four  reasons  why  managers  do 
not  use  models  more  widely: 

•  Good  models  are  hard  to  find.  Models,  which  accurately  depict 
management  control  and  environmental  variables,  are  difficult  to  build. 

•  Good  parameterization  is  even  harder.  Metrics  and  accurate,  timely  data 
are  a  mandatory  requirement.  Complex,  high  quality  design  work  is 
required  and  expensive. 

•  Managers  don't  understand  the  models.  A  manager  is  responsible  for  his 
decisions  and  will  not  trust  the  output  from  a  model  he  does  not  fully 
comprehend.  Often,  a  manager  will  choose  a  simpler  analysis  that  he  can 
grasp  over  a  model  whose  assumptions  are  partially  hidden  and  contain 
complex  statistical  manipulations. 

•  Most  models  are  incomplete.  An  incomplete  model  can  pose  a  serious  risk 
to  optimization  analysis. 

C.  SOFTWARE  DEVELOPMENT  LITERATURE 

Since  1991,  the  LMDSS  development  effort  has  faced  more  than  its  share  of 
obstacles.  LMDSS  meets  the  Standish  Group's  criteria  for  a  challenged  software  project 
because  it  is  over  budget,  has  exceeded  its  original  time  estimate,  and  contains  fewer 
features  than  originally  specified  (Standish  Group,  1999).  Research  is  now  available 
which  can  help  managers,  developers,  and  clients  better  understand  the  risks  and  classic 
mistakes  of  the  software  development  process.  Research  literature  was  selected  to  show 
how  prevalent  the  LMDSS  situation  is  among  other  organizations  and  to  provide  needed 
guidance  to  software  developers  seeking  to  bring  such  projects  under  control.  Proven 
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fundamental  strategies  are  presented  which  can  significantly  assist  software  development 
teams  in  their  efforts  to  rapidly  achieve  their  objectives  within  realistic  cost  constraints. 

1.  The  Software  Crisis 

The  Standish  Group's  (1999)  research  has  shown  only  nine  percent  of  software 
development  projects  within  larger  companies  are  completed  on  time  and  on  budget.  The 
average  cost  overrun  for  these  projects  was  178  percent.  In  1995,  American  companies 
spent  an  additional  $59  billion  to  complete  software  projects  that  exceeded  their  original 
time  estimates.  The  average  project  overrun  is  222  percent  of  the  original  time  estimate. 
Excessive  schedule  pressure  occurs  in  75  percent  of  all  large  software  projects  and  close 
to  100  percent  of  all  very  large  projects  (Jones,  1994).  The  average  software  developer  in 
the  United  States  works  48  to  50  hours  a  week  (Krantz,  1995). 

In  this  environment,  it  isn't  surprising  that  general  job  satisfaction  of 
software  developers  has  dropped  significantly  in  die  last  15  years 
(Zawacki  1993),  and  at  a  time  when  the  industry  desperately  needs  to  be 
recruiting  additional  programmers  to  ease  the  schedule  pressure, 
developers  are  spreading  the  word  to  their  younger  sisters,  brothers,  and 
children  that  our  field  is  no  fun  anymore  (McConnell,  1996,  pg.  xviii). 

Survey  results  show  there  are  plenty  of  organizations  struggling  to  get  software 
development  projects  under  control.  Most  are  trying  to  overcome  the  same  people, 
process,  and  technology  issues  faced  by  NAVAIR's  LMDSS  development  team.  The  first 
step  to  improving  the  software  development  process  is  to  identify  the  ineffective 
practices  and  classic  mistakes.  The  second  step  involves  a  return  to  effective,  proven 
fundamental  software  development  practices.  The  third  step  is  to  manage  identified  risks 
to  avoid  major  setbacks.  (McConnell,  1996) 

2.  Classic  Mistakes  Leading  to  the  Ten  Most  Serious  Risks  to  Software 
Development 

McConnell  (1996)  identified  36  classic  mistakes  in  the  software  development 
process,  which  can  lead  to  a  failed  program.  These  classic  mistakes  are  described  in 
Appendix  A.  Based  on  research  results  involving  Software  Productivity  Research  (SPR) 


13 


assessments.  Capers  Jones  (1994)  reported  the  mistakes  that  contribute  the  most  to  the 
failure  of  software  projects  as: 

•  Lack  of  historical  software-measurement  data 

•  Rejection  of  accurate  measurement  estimates 

•  Failure  to  use  automated  estimating  and  planning  tools 

•  Excessive,  irrational  schedule  pressure 

•  Creeping  (new  and  unanticipated)  user  requirements 

•  Failure  to  monitor  progress  and  to  perform  formal  risk  management 

•  Failure  to  use  design  reviews  and  code  inspections 

From  a  total  sample  size  of  365  respondents  representing  8,380  software 
applications.  The  Standish  Group  (1999,  p.  5)  conducted  four  focus  groups  and 
interviews  with  various  IT  managers.  Based  on  the  previously  discussed  definition  of  a 
challenged  software  project,  they  identified  ten  factors,  that  contribute  to  such  a  project: 

•  Lack  of  user  input 

•  Incomplete  requirements  and  specifications 

•  Changing  requirements  and  specifications 

•  Lack  of  executive  support 

•  Technology  incompetence 

•  Lack  of  resources 

•  Unrealistic  expectations 

•  Unclear  objectives 

•  Unrealistic  time  frames 

•  New  technology 
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Once  the  mistakes  and  ineffective  practices  have  been  identified,  they  must  be 
corrected  and  replaced  with  proven  management,  technical  and  quality  assurance  (QA) 
fundamental  practices.  After  completing  a  study  of  ten  "best  projects,"  one  researcher 
concluded. 


If  there  was  one  high-level  finding  that  stood  out,  it  is  that  best 

projects  get  to  be  best  based  on  fundamentals.  All  of  us  know  the 

fundamentals  for  good  software-the  difference  is  that  most  projects  don’t 

do  them  nearly  so  well  and  then  get  into  trouble.  (Hetzel,  1993,  p.  151) 

3.  Management  Fundamentals 

Management  fundamentals  involve  estimating  the  size  and  characteristics  of  the 
project  (including  software  functionality  and  complexity),  assigning  adequate  resources, 
creating  a  plan,  and  monitoring  and  directing  the  plan  to  stay  on  course.  Effective 
estimation  strategies  are  essential  to  developing  a  realistic  plan  upon  which  both 
managers  and  developers  can  depend.  (McConnell,  1996)  The  best  projects  are 
characterized  by  strong  up-front  planning  to  define  tasks  and  schedules  (Hetzel,  1993). 
A  software  development  project  should  include: 

•  Estimation  and  scheduling 

•  Determining  how  many  people  to  have  on  the  project  team,  what  technical 
skills  are  needed,  when  to  add  people,  who  the  people  will  be 

•  Deciding  how  to  organize  the  team 

•  Choosing  which  lifecycle  model  to  use 

•  Managing  risks 

•  Making  strategic  decisions  such  as  how  to  control  the  product's  feature  set 

and  whether  to  buy  or  build  pieces  of  the  product  (McConnell,  1996,  p.56) 

Once  the  plan  is  developed,  it  has  to  be  monitored  to  ensure  all  cost,  schedule, 
and  quality  milestones  are  met.  Fundamental  management  tracking  activities  include: 
task  lists,  status  meetings,  status  reports,  milestone  reviews,  budget  reports,  and 
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management  by  walking  around.  Fundamental  technical  tracking  events  include 
technical  audits,  technical  reviews,  peer  reviews,  and  quality  gates  to  ensure  functional 
objectives  are  met  at  the  appropriate  milestone.  (McConnell,  1996)  Not  implementing 
tracking  procedures  properly  can  have  disastrous  effects.  Jones  (1995,  p.  87)  reported 
that,  "Software  progress  monitoring  is  so  poor  that  several  well-known  software  disasters 
were  not  anticipated  until  the  very  day  of  expected  deployment." 

If  you  don't  track  a  project,  you  can’t  manage  it.  You  have  no  way  of 
knowing  whether  your  plans  are  being  carried  out  and  no  way  of  knowing 
what  you  should  do  next.  You  have  no  way  of  monitoring  risks  to  your 
project.  Effective  tracking  enables  you  to  detect  schedule  problems  early, 
while  there's  still  time  to  do  something  about  them.  (McConnell,  1996,  p. 

57-58) 

Collecting  metrics  is  a  critical  management  requirement  to  track  and  analyze 
software  quality  and  productivity.  Most  projects  collect  measurements  of  cost  and 
schedule  targets,  but  such  data  doesn’t  provide  the  insight  needed  to  reduce  program 
costs  or  shorten  time  schedules.  A  knowledge  of  specific  metrics  needed  to  analyze 
project  status,  quality,  and  productivity  is  required  to  know  whether  or  not  a  project's 
development  speed  is  improving  or  degrading.  (McConnell,  1996) 

The  importance  of  metrics  in  planning  and  estimating  is  reinforced  by  Jones 
(1996,  p.  104): 

Software  failures  are  caused  primarily  by  errors  and  poor  judgement 
on  the  part  of  managers,  executives,  and  clients  -  not  errors  made  by  the 
technical  teams.  The  root  cause  of  these  failures  is  the  lack  of  accurate 
measurement  data,  which  blinds  management  and  clients  to  what  is 
possible  and  what  might  be  possible. 

4.  Technical  Fundamentals 

Requirements  management  involves  gathering  and  documenting  user 
requirements,  tracking  the  project  design  and  code,  and  managing  all  changes  for  the 
entire  project  (McConnell,  1996).  A  Standish  Group  (1994)  survey  of  over  8000 
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software  projects  found  lack  of  user  input,  incomplete  requirements,  and  changing 
requirements  were  the  top  three  reasons  for  late,  over  budget  projects  with  less  than 
anticipated  functionality.  The  fundamentals  of  requirements  management  are: 

•  Requirements  analysis  methodologies  including  structured  analysis,  data 
structured  analysis,  and  object  oriented  analysis 

•  System  modeling  practices  such  as  class  diagrams,  dataflow  diagrams, 
entity-relationship  diagrams,  data  dictionary  notation,  and  state  transition 
diagrams 

•  Communication  practices  such  as  Joint  Application  Development,  user- 
interface  prototyping,  and  general  interview  practices 

•  The  relationships  between  requirements  management  and  the  different 
lifecycle  models  including  evolutionary  prototyping.  (McConnell,  1996,  p. 
62) 

Research  has  shown  that  the  cost  of  reworking  software  is  smaller  by  a  factor  of 
50  to  200  in  the  earlier  phases  of  the  development  cycle  than  waiting  until  the 
construction  or  maintenance  phases  to  correct  them.  This  puts  a  high  emphasis  on  early 
error  detection  and  getting  requirements  correct  the  first  time.  (Boehm  and  Papaccio, 
1988) 

Software  configuration  management  is  defined  as  the  “methodical  storage  and 
recording  of  all  software  components  and  deliverables  as  projects  are  under 
development”  (Jones,  1994,  p.  556).  It  includes  evaluating  proposed  software  changes, 
monitoring  changes,  updating  planning  documents,  keeping  records  of  all  previous 
versions,  and  tracing  backwards  to  original  requirements.  Automated  tools  are  available 
which  can  help  developers  synchronize,  cross-reference,  integrate,  and  update 
specifications,  graphics,  user  information,  source  code,  defect  repairs,  and  test  cases. 
Configuration  management  is  a  requirement  for  military  software  and  often  represents  a 
significant  cost  factor.  (Jones,  1994) 
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Poor  configuration  management  can  allow  developers  to  write  code  that  is 
incompatible  with  code  written  by  other  team  members  resulting  in  expensive  rework. 
Jones  (1994)  reported  that  30  percent  of  U.S.  software  producers  had  no  configuration 
control  automation  of  any  kind.  About  30  percent  had  some  form  of  source  code 
automation,  but  no  automation  for  documentation,  test  cases,  and  defect  tracking.  For  a 
nominal  10,000  function  point  project,  Jones  reported  an  added  cost  of  168  man-months 
for  projects  that  failed  to  use  automation.  For  large  projects,  he  reported  configuration 
control  to  be  a  critical  path  item. 

5.  Quality  Assurance  Fundamentals 

a)  The  Cost  of  Software  Defects 

Research  has  overwhelming  shown  that  identifying  and  correcting 
software  defects  early  in  the  development  process  can  generate  significant  savings  in 
both  time  and  money.  Every  hour  spent  on  defect  prevention  will  reduce  a  project's 
rework  time  by  three  to  ten  hours.  After  surveying  approximately  4000  projects,  Jones 
reported  one  of  the  most  common  causes  of  schedule  overrun  was  poor  quality  software. 
(Jones,  1994)  Another  survey  conducted  at  59  government  and  industry  software  sites 
examined  296  software  projects  and  found  that  quality  assurance  was  a  significant 
problem  at  63  percent  of  the  sites  assessed  (Kitson  and  Masters,  1993). 

Late  projects  are  particularly  vulnerable  to  shortening  the  QA  process  in 
response  to  pressures  to  meet  deadlines.  Surveys  have  found  that  software  products 
developed  under  excessive  schedule  pressure  have  reported  up  to  4  times  the  number  of 
normal  defects  (Jones,  1994). 

b)  Strategies  in  Achieving  Quality  Software 

The  most  common  strategy  for  finding  software  defects  is  testing.  Testing 
should  include  unit  testing,  where  the  developer  checks  his  own  code,  and  system  testing, 
where  an  independent  tester  checks  the  application  to  verify  it  works  as  advertised.  The 
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best  way  to  implement  testing  is  to  conduct  it  in  a  manner  where  errors  can  be  identified 
and  corrected  as  early  as  possible.  (McConnell,  1996) 

A  key  element  of  the  QA  process  is  formal  technical  reviews.  Technical 
reviews  include  formal  design  reviews,  walkthroughs,  internal  code  reading,  and 
inspections. 

Formal  design  reviews  provide  feedback  to  management  concerning  the 
status  of  the  software  project  under  development.  They  typically  occur  at  the  end  of  the 
various  development  phases  and  based  on  their  results,  management  decides  whether  or 
not  the  product  is  ready  to  proceed  to  the  next  phase.  - 

Walkthroughs  are  meetings  where  two  or  more  developers  form  a  peer 
review  team  to  inspect  specific  sections  of  a  co-workers  code  with  the  purpose  of 
improving  quality.  Walkthroughs  are  essential  to  the  QA  process  because  they  can  be 
used  to  detect  software  defects  well  before  normal  testing  is  accomplished.  (McConnell, 
1996)  Documenting  these  reviews  is  accomplished  through  a  walkthrough  summary 
report  that  contains  what  was  reviewed,  who  reviewed  it,  and  what  findings  and 
conclusions  were  reached. 

Code  reviews  are  more  formal  than  walkthroughs  and  usually  limited  to 
inspections  of  only  code.  A  code  review  consists  of  a  developer  passing  his  work  to  two 
or  more  reviewers.  The  reviewers  read  the  code  and  report  any  discovered  defects  to  the 
author  of  the  code.  (McConnell,  1996)  A  study  at  NASA's  Software  Engineering 
Laboratory  by  Card  (1987)  found  that  internal  code  reading  procedures  detected  twice  as 
many  defects  per  hour  of  effort  as  testing. 

Software  inspections  are  technical  reviews  where  each  design  component 
is  inspected  at  the  requirements,  preliminary  design,  detailed  design,  and  code  phases. 
Inspections  use  trained  peer  teams  of  three  to  eight  people  with  each  person  playing  a 
specific  role.  A  team  leader  hands  out  the  work  to  be  inspected  before  the  formal 
meetings.  Reviewers  examine  the  work  using  checklists  and  arrive  at  the  team  meetings 
with  their  results.  At  team  meetings  defects  are  identified  and  a  plan  of  action  for 
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correcting  them  is  developed.  Throughout  the  inspection  process  data  on  all  defects, 
hours  spent  correcting  them,  and  hours  devoted  to  inspections  are  collected  to  assist  in 
the  improvement  of  the  software  development  process.  (McConnell,  1996)  A  study  by 
Russell  (1991)  found  that  for  large  programs,  each  hour  spent  on  inspections  avoided  an 
average  of  thirty-three  hours  of  maintenance.  He  also  noted  that  inspections  were  up  to 
twenty  times  more  efficient  than  testing. 

6.  Risk  Management 

Many  of  the  proven,  fundamental  software  development  practices  described  are 
intertwined  and  rely  on  each  other  to  be  successfully  implemented.  Jones  (1996,  p.  103) 
states. 


A  well-formed  risk  analysis  and  milestone-tracking  program  for 
software  projects  depends,  quite  simply,  on  successful  completion  of 
formal  design  and  code  inspections. 


The  purpose  of  a  formal  risk  management  program  is  to  identify,  address,  and 
minimize  sources  of  risk  before  they  create  significant  obstacles  to  a  software  project. 
The  elements  of  risk  management  include  risk  assessment  and  risk  control  (McConnell, 
1996). 


a)  Risk  Assessment 

Risk  assessment  is  made  up  of  risk  identification,  risk  analysis,  and  risk 
prioritization.  Risk  identification  simply  identifies  the  factors  that  represent  a  risk  to  a 
software  project’s  successful  completion.  The  most  common  Management  Information 
System  (MIS)  software  risks  identified  in  the  literature  are  found  below: 

•  Creeping  user  requirements 

•  Excessive  schedule  pressure 

•  Low  quality 

•  Cost  overruns 
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Inadequate  configuration  control  (Jones,  1994,  p.  29) 


Risk  analysis  involves  determining  the  impact  of  each  risk  identified.  Risk 
analysis  can  involve  estimating  probabilities,  the  size  of  losses,  and  the  levels  of 
exposure  in  time  periods  such  as  weeks  or  months.  Risk  analysis  may  also  involve  the 
use  of  performance  and  cost  models,  decision  analysis,  and  quality  factor  analysis. 

Risk  prioritization  follows  the  identification  and  analysis  phases  with  the 
development  of  a  priority  list  of  risks  so  risk  reduction  efforts  can  be  focused  on  the 
greatest  threats.  Boehm  and  Papaccio  (1988,  p.  1466)  noted, 

80  percent  of  the  rework  costs  typically  result  from  20  percent  of  the 
problems... The  major  implication  of  this  distribution  is  that  software 
verification  and  validation  activities  should  focus  on  identifying  and 
eliminating  the  specific  high-risk  problems  to  be  encountered  by  a 
software  project,  rather  than  spreading  their  available  early-problem- 
elimination  effort  uniformly  across  trivial  and  severe  problems. 

b)  Risk  Control 

Once  risks  have  been  identified,  measured,  and  prioritized,  they  have  to  be 
controlled  and  minimized.  Risk  control  activities  include  risk  management  planning,  risk 
resolution,  and  risk  monitoring. 

The  purpose  of  risk  management  planning  is  to  develop  a  written  plan  to 
address  the  highest  priority  risks  identified  earlier.  The  plan  can  be  as  simple  as  a 
paragraph  for  each  risk  detailing  the  who,  what,  where,  how,  and  why  of  each  risk's 
management  strategy.  The  plan  should  describe  how  risk  management  strategies  will  be 
monitored,  how  eliminated  risks  will  be  closed  out,  and  what  new  risks  have  been 
identified.  (McConnell,  1996) 

Risk  resolution  is  the  implementation  of  the  risk  management  plan  to 
avoid,  reduce,  transfer,  eliminate  or  control  identified  high  priority  risks.  Prototypes, 
simulations,  and  benchmarking  may  be  used  in  this  phase  to  investigate  and  resolve  risks. 
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Risk  monitoring  also  involves  implementing  the  risk  management  plan  by 
tracking  risk  management  actions  and  documenting  identified  emerging  risks.  Risk 
monitoring  procedures  often  include  milestone  tracking,  risk  reassessments,  and 
documenting  corrective  action. 

7.  Research  Recommendations  for  Avoiding  Disaster 

Based  on  his  years  of  researching  hundreds  of  software  projects,  Jones  (1996,  p. 
104)  supplements  the  proven  practices  previously  described  with  some  additional 
recommendations  to  achieving  success  in  software  development  efforts: 

•  Look  at  the  actual  results  of  similar  projects 

•  Make  planning  and  estimating  formal  activities 

•  Plan  for  and  control  creeping  requirements 

•  Use  formal  inspections  as  milestones  for  tracking  project  progress;  and 

•  Collect  accurate  measurement  data,  during  the  current  project,  to  use  with 
future  projects 
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m.  BACKGROUND  INFORMATION 


A.  DEVELOPMENT  HISTORY 

1.  Background  Information 

The  end  of  the  Cold  War  created  a  new  world  order  with  the  United  States  as  the 
only  surviving  superpower.  After  the  fall  of  the  Berlin  Wall,  the  American  public  and 
members  of  Congress  anticipated  massive  cuts  in  the  military.  An  immediate  sense  of 
increased  security  prevailed,  resulting  in  demands  for  heavy  reductions  in  defense 
spending.  The  term  “peace  dividend”  was  coined  and  popularized  as  it  was  felt  that  the 
savings  generated  would  be  applied  to  domestic  programs  and  to  help  lower  the  deficit. 
Over  the  next  several  years,  the  defense  budget  was  drastically  reduced,  to  the  point 
where  the  fiscal  year  1999  defense  budget  represented  a  39  percent  drop  from  spending 
levels  in  the  1980s.  The  defense  budget  is  now  equal  to  just  3.2  percent  of  our  gross 
national  product,  a  level  not  seen  since  the  1930s. 

The  magnitude  of  these  reductions  was  felt  throughout  the  Department  of 
Defense.  Military  bases  were  closed,  ships  were  decommissioned,  entire  air  wings  were 
disestablished  and  thousands  of  fully  trained  troops  were  discharged.  Acquisition 
programs  that  had  been  funded  for  years  were  severely  restricted  or  canceled  out-right. 
The  only  thing  in  the  military  that  wasn’t  cut  back  was  the  operational  tempo.  Fewer 
troops  were  expected  to  fulfill  the  same  requirements  with  older  and  less  reliable 
equipment.  Operating  and  support  costs  skyrocketed  as  aging  aircraft,  tanks  and  ships 
were  pushed  beyond  their  expected  useful  life  in  order  to  meet  operational  needs. 

Program  managers,  trying  to  do  more  with  less,  were  forced  to  look  closely  at 
their  programs  and  examine  all  expenditures  in  an  effort  to  eliminate  unnecessary  cost 
drivers.  For  most  program  managers,  this  was  a  tedious  and  time-consuming  task;  for 
some  it  was  simply  impossible.  The  sheer  volume  of  data,  which  were  distributed  in 
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technically  diverse  and  geographically  separated  databases,  made  the  job  prohibitively 
expensive. 

Analysts  at  NAVAIR  4.0  and  NAVICP,  formerly  the  ASO,  were  charged  with  the 
task  of  assessing  the  logistics  health  of  aviation  programs,  focusing  primarily  on 
readiness,  supportability,  and  cost.  This  involved  the  time-consuming  manual  efforts  of 
pouring  though  the  available  data  in  an  effort  to  minimize  costs.  After  several  months  of 
effort,  they  concluded  that  the  information  systems  available  to  them  were  inadequate  for 
the  job.  If  they  were  to  meet  the  demands  of  cost  reductions  through  analysis  of  the 
disparate,  poorly  maintained  databases,  program  managers  sorely  needed  a  tool  that 
could  provide  a  central  data  repositoiy  and  support  the  analysis  of  the  data. 

With  this  in  mind,  steps  were  taken  within  NAVAIR  to  assess  the  feasibility  of 
designing  and  implementing  a  DSS  specifically  to  support  logistics  decisions.  In  this 
manner  the  concept  of  LMDSS  was  founded. 

2.  LMDSS  Fundamental  Structure 

As  stated  above,  the  driving  requirement  behind  the  push  for  the  development  of  a 
DSS  for  logistics  management  was  the  need  for  a  tool  to  facilitate  measurements  to  use 
in  the  planning  and  identification  of  targets  for  cost  reduction.  The  original  vision 
statement  for  LMDSS  required  the  detailed  assessment  and  development  of  a  DSS  which 
would  be  the  primary  tool  for  APML/Weapon  System  Managers  (WSM)  to  achieve  a 
“continuous,  measurable  reduction  in  life-cycle  costs  while  maintaining  operational 
readiness  and  sustainability.”  (Naval  Air  Systems  Command,  1993). 

The  specific  goals  of  the  initial  LMDSS  program  as  designed  within  the  Naval 
Aviation  Logistics  Data  Analysis  (NALDA)  system  were  as  follows: 

•  To  be  the  primary  DSS  to  achieve  cost-effective  logistics  management 

•  Improve  more  timely  (daily)  receipt  of  AV3M  and  configuration  data 

•  Create  cost-effective  consolidation  of  central,  upline  AV-3M  data  systems 
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•  Provide  the  ability  to  access  centralized  Fleet-wide,  near  real  time 
operational  and  readiness  data  from  NALDA  in  accordance  with  DoD  data 
security  regulations.  (Naval  Air  Systems  Command,  Program  Manager's 
Charter,  1997) 


In  order  to  meet  these  objectives,  it  was  essential  to  integrate  vast  quantities  of 
diverse  data  contained  in  multiple  legacy  stovepipe  databases  into  one  central  repository. 
The  LMDSS  requirement  document  (LMDSS  Requirements  Document/Statement  of 
Work,  1993)  also  identified  other  key  features  as  being  essential  to  a  successful  program. 
These  included: 

•  Timely,  precise  responses  to  queries  independent  of  type  data  or 
type/model/series  (TMS) 

•  A  user  friendly  system  that  did  not  require  extensive  computer  knowledge 
or  training 

•  Ease  of  access  to  the  system  as  well  as  maximizing  the  eligible  number  of 
personnel  who  could  access  the  system 

•  Detailed  statistical  packages  embedded  in  the  program  to  provide 
projections  of  outcome  based  upon  changes  to  maintenance  and  supply 
parameters 

•  Both  a  structured  modular  approach  to  data  recovery  and  an  ad  hoc 
Structured  Query  Language  (SQL)  capability 

•  Assist  tools  to  facilitate  easy  development  of  queries 

•  Integration  of  the  depot  level  AV3M  data  and  the  Naval  Depot  Logistics 
Management  System  (NDLMS)  into  the  NALDA  phase  n  existing 
database  structure 

•  Graphical  User  Interface  (GUI)  capabilities  designed  to  produce 
presentation  quality  graphics  on  data  obtained  from  queries 

•  Inclusion  of  an  Evaluation  and  Selection  of  Alternatives  Module  (ESM) 
based  upon  return  on  investment  principles 
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•  Inclusion  of  implementation  and  status  tracking  modules  to  document  the 
actual  implementation  of  actions  recommended  in  the  ESM  module  and  to 
establish  an  audit  trail 

According  to  the  initial  Statement  of  Work  (SOW),  rapid  prototyping  was  the 
software  development  method  used  to  develop  the  initial  prototype.  The  SOW  also 
required  the  LMDSS  software  development  teams  to  maintain  maximum  flexibility  and 
“react  with  rapid  turn  around  to  additional  requirements  and  changes  to  requirements 
levied  by  the  LMDSS  functional  design  team”  (LMDSS  Requirements  Document/SOW, 
1993). 

At  this  time,  the  process  for  Major  Acquisition  of  Information  Systems  (MAIS) 
was  still  a  relatively  new  concept.  Exacting  measures  and  legislation  were  not  in  place  to 
help  identify  means  of  formally  structuring  software  development  processes.  Most 
legislation  was  developed  in  response  to  unforeseen  difficulties  arising  from  haphazard 
and  unstructured  software  development  processes. 

B.  LMDSS’  ROLE  TN  SUPPORTING  NAVAL  AVIATION  FOR  THE  21 ST 

CENTURY 

1.  LMDSS  Within  the  NALDA  System 

LMDSS  is  currently  a  sub-component  of  the  NALDA  Phase  II  project.  NALDA 
has  been  operational  since  the  early  1980s,  although  LMDSS  was  not  formally  initialized 
until  the  early  1990s.  NALDA  is  the  Navy  and  Marine  Corps  central  aviation 
maintenance  and  logistics  Automated  Information  System  (AIS).  Its  mission  is  to 
provide  Integrated  Logistics  Support  (ILS),  engineering  functions  and  achieve  readiness 
and  affordability  process  improvements  through  the  implementation  of  advanced  data 
collection,  compilation  and  management  techniques  (Naval  Air  Systems  Command, 
Mission  Need  Statement,  1997).  Other  applications  that  fall  under  the  umbrella  of  the 
NALDA  system  include  logistics  planning,  management,  administrative,  budgeting  and 
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resource  allocation  for  aviation  weapon  systems  and  related  support  equipment.  For  a 
more  detailed  description  of  NALDA II,  refer  to  Appendix  B. 

NALDA  Phase  I  currently  operates  on  an  AMDAHL  5995  mainframe  located  at 
the  Defense  MegaCenter  in  Mechanicsburg,  Pennsylvania,  but  it  is  in  the  process  of 
being  phased  out.  This  system  is  redundant,  consists  of  outdated  stovepipe  systems  with 
restricted  functionality,  and  does  not  meet  the  requirements  as  set  forth  in  current 
doctrine.  In  addition,  many  of  the  older  legacy  systems  are  not  Year  2000  (Y2K) 
compliant  and  the  Defense  Information  Systems  Agency  (DISA)  has  mandated  all  such 
systems  be  made  Y2K  compliant  or  be  shut  down  by  15  March  1999.  The  NAT  DA 
system  is  scheduled  to  be  completely  converted  to  an  IBM  RS/6000  environment  using 
the  Advanced  Interactive  executive  (AIX)  operating  system  and  employing  an  Oracle  7.3 
relational  database  management  system  by  the  end  of  second  quarter  FY99.  The  current 
mainframe  system,  a  Reduced  Instruction  Set  Computing  (RISC)  SP  machine,  is 
operational  but  multiple  unforeseen  difficulties  with  the  data  loads  has  caused  delays  in 
the  schedule.  Parallel  operations  were  planned  to  be  in  effect  for  a  three  month  period 
from  January  to  March  1999  in  order  to  minimize  the  risk  of  decreased  performance 
while  identifying  glitches  in  the  new  system.  During  this  time  period,  NALDA  I  is 
scheduled  to  run  on  the  server  at  Mechanicsburg  while  NALDA  IT  runs  concurrently  on 
the  server  at  NAS  Patuxent  River.  As  of  May  1999,  the  dual  system  approach  was 
extended  to  continue  through  the  end  of  the  fiscal  year. 

2.  LMDSS  Within  the  Department  of  Navy  (DON)  Chief  Information 

Officer  (CIO)/GlobaI  Combat  Support  Structure  (GCSS) 

The  Office  of  Secretary  of  Defense  Certification  (December  21,  1995)  for  the 
NALDA  system  defined  the  NALDA  migration  strategy.  The  NALDA  n  program,  when 
completed,  will  ensure  compliance  with  the  Department  of  the  Navy  Information 
Technology  Strategic  Plan  which  requires  logistic  support  applications  provide  a 
seamless  blend  of  operational  and  administrative  information  to  all  users  (Naval  Air 
Systems  Command,  Program  Manager's  Charter,  1997).  In  addition,  the  plan  was 
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structured  in  accordance  with  the  Technical  Architecture  Framework  for  Information 
Management  because  it  established  a  long-range  goal  of  open  systems  architecture.  It 
also  met  Defense  Information  Infrastructure  (DII)  Navy  Common  Operating  Environment 
(COE)  criteria.  NALDA  Phase  II  plans  meet  the  Level  5  DII  COE  standard  as  defined  in 
the  Integration  and  Runtime  Specification  which  enables  it  to  be  interoperable  with  other 
Level  5  or  higher  DII  COE  systems  like  the  Joint  Maritime  Command  Information 
System  (JMCIS)  or  the  Global  Combat  Support  System  (GCCS).  (Naval  Air  Systems 
Command,  Operational  Requirements  Document,  1997) 

C.  CURRENT  LMDSS  ENVIRONMENT 

1.  Initial  Prototype  Development 

The  LMDSS  Statement  of  Work  was  developed  in  1993.  The  initial  prototype 
was  completed  in  1994.  It  used  the  current  NALDA  IBM  RISC  System/6000  Model  970 
at  ASO  Philadelphia  and  other  regional  machines  located  at  NADEP  and  Naval  Air  War 
Center  (NAWC).  The  relatively  new  X-windows  specifications,  as  developed  by  the 
Massachusetts  Institute  of  Technology,  were  incorporated  as  the  standard  for  GUIs.  X- 
client  and  X-server  structures  were  used.  All  programming  on  the  initial  prototype  was 
done  in  Ada  programming  language.  AIX  operating  systems  which  are  IBM’s  version  of 
the  single  user  version  of  MULT  “ICS”  (UNIX),  were  used  exclusively  and  Transmission 
Control  Protocol/Intemet  protocol  (TCP/IP)  was  file  only  protocol  used  in  NAVAIR's 
wide  area  network  (WAN)  connectivity  plans.  In  addition,  the  Oracle  relational 
database  concept  was  the  basis  for  the  organizational  matrices  of  the  LMDSS  database. 

The  prototype  worked  well,  but  several  of  the  initial  requirements  were  not  met. 
The  UNIX  operating  system  was  unfriendly  to  all  but  the  most  highly  trained  computer 
personnel.  The  X-client  and  X-server  implementation  mandated  the  usage  of  a  direct 
*  local  area  network  for  personal  computer  accessibility.  For  those  personnel  on  a  WAN, 
the  X-windows  required  a  56KBit/sec  connectivity  rate.  Although  that  rate  is  fairly 
common  in  1999,  five  years  ago  it  was  difficult  and  expensive  to  obtain  reliable  T-l  line 
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capability.  Only  the  PMAs  and  APMLs  located  at  NAS  Patuxent  River  could  get  access 
to  LMDSS.  The  contractors  and  analysts  hired  by  PMAs  for  the  majority  of  their  data 
analysis  did  not  have  access  to  LMDSS  because  they  were  not  part  of  NAS  Patuxent 
River  WAN.  At  this  point,  LMDSS  was  still  considered  a  logistics  management  tool  for 
high  level  logisticians,  and  according  to  later  surveys,  most  of  them  were  simply  not 
using  it  (Moore  and  Snyder,  1998). 

2.  Transition  to  Web  Based  Format 

In  1995,  NAVAER  determined  the  detriments  of  the  existing  prototype  did  not 
meet  the  requirements  as  set  out  in  the  SOW.  The  decision  was  made  to  abandon  the 
UNIX/X-windows  environment  in  order  to  develop  a  web-based  system  with  Common 
Graphics  Interface  (CGI)  scripting.  The  Ada  programming  language  was  abandoned  for 
HyperText  Markup  Language  (HTML)  and  Practical  Extraction  and  Report  Language 
(PERL)  because  it  was  not  suited  for  a  major  database  management  system.  Although 
these  decisions  directly  impacted  major  software  development  issues,  the  user  group  was 
neither  expanded  nor  queried  for  inputs.  A  thorough  risk  analysis  to  determine  the  impact 
of  these  decisions  was  also  not  conducted  and  documented.  The  project  fell  behind  the 
scheduled  baseline  and  costs  exceeded  earlier  estimates.  As  shown  in  the  review  of  the 
software  development  literature,  this  is  a  common  occurrence  within  the  software 
industry.  The  Standish  Group  (1999)  reported  only  nine  percent  of  software  developed 
by  larger  companies  come  in  on-time  and  on-budget.  Jones  (1994)  reported  that  most 
software  development  projects  are  cancelled  because  they  fail  to  properly  conduct 
requirements  analysis,  identity  user  needs,  and  conduct  risk  analysis/reduction 
assessments.  As  a  result,  instead  of  anticipating  problems,  the  LMDSS  team  fell  into  the 
practice  of  reacting  to  events  as  difficulties  arose. 

3.  Integration  into  NALDA  Program 

It  was  shortly  thereafter  that  the  decision  was  made  to  make  LMDSS  the  basic 
interface  to  all  NALDA  data.  The  original  LMDSS  contract  had  been  awarded  to  a  small 
software  development  firm.  It  quickly  became  apparent  the  firm  did  not  have  the 
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expertise  in  HTML,  CGI  Scripting,  ActiveX  and  data  warehousing  to  meet  the  needs  of 
the  revised  program  specifications.  When  their  contract  was  up,  a  major  programming 
contractor  was  selected  to  replace  the  small  business.  Several  of  the  firm's  former 
employees  stayed  on  as  Federal  Government  Service  workers.  During  this  transition  the 
LMDSS  programmer  pool  was  reduced  from  over  thirteen  full  time  programmers,  to  only 
three  who  were  concurrently  seeking  other  employment.  It  is  presumed  that  a  large  chunk 
of  corporate  knowledge  was  lost  in  this  contractor  change,  as  not  all  programmers  were 
hired  on  by  the  new  company. 
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Figure  1 NALDA  Upline  Application 

LMDSS  was  changed  once  again  from  the  basic  interface  for  all  of  NALDA,  to  its 
current  status  as  just  the  component  of  the  NALDA  II  system  that  provides  access  to  the 
reliability  and  maintenance  portion  of  the  NALDA  n  database  structure.  Figure  1 
displays  the  current  NALDA  upline  configuration  and  includes  the  new  functional 
relationship  of  LMDSS  within  the  NALDA  II  system  (NAVAIR  3.6,  1999).  LMDSS 
does  not  do  configuration  management  (CM),  although  it  helps  track  CM  data.  When  it 
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first  was  produced  in  1994,  it  was  hoped  by  NAVAIR  that  LMDSS  would  be  the  end-all, 
solve-all  for  any  questions  concerning  reliability,  readiness,  cost,  and  configuration 
management  for  the  Navy's  aircraft.  This  is  no  longer  the  case;  however,  when  the 
development  of  LMDSS  is  complete,  it  will  have  all  the  detailed  data  from 
approximately  2.5  million  maintenance  action  records/month.  Currently,  NAVAIR  has 
archived  32  months  of  maintenance  data  online  and  1 1  years  of  flight  data  (Interview 
with  Mr.  Joe  Joseph,  1998). 

Unfortunately,  under  the  new  NALDA  II  Acquisition  Program  Baseline  (APB), 
LMDSS  developers  were  forced  to  meet  the  deadlines  established  for  the  Milestone  HI 
Decision  Authority.  The  first  increment,  which  consisted  of  the  consolidation  of  the 
configuration  management  and  AV-3M  systems,  was  to  be  completed  by  February  1998. 
In  the  process  of  transitioning  from  one  software  language  and  platform  to  another  and  to 
meet  the  goals  of  the  NALDA  program,  many  DSS  items  were  discarded.  Graphics 
capabilities  and  some  of  the  statistical  forecasting  and  modeling  modules  were  scrapped. 

What  makes  the  current  application  of  LMDSS  useful  is  it’s  summary  tables  that 
can  provide,  for  example,  the  top  five  items  in  the  last  five  years  with  the  worst  reliability 
performances  for  the  maritime  patrol  aircraft  (P-3)  community.  Another  strength  is  its 
ability  to  drill  down  from  the  summary  data  to  specific  part  numbers  and  locations.  Even 
with  its  reduced  capabilities  caused  by  the  multiple  changes  in  software  format  and 
program  structure,  the  remaining  capabilities  will  enable  NAVAIR  program  managers  to 
identify  areas  to  save  money  so  they  can  use  the  funds  to  purchase  more  important  items 
that  are  not  funded.  In  short,  doing  more  with  less. 

4.  Current  Program  Funding 

Funding  reductions  have  had  the  most  detrimental  impact  on  the  LMDSS 
program.  LMDSS  received  its  initial  charter  prior  to  the  finalization  of  the  plans  for 
NALDA  Phase  II,  although  funding  for  LMDSS  was  and  still  is  received  mainly  under 
the  NALDA  system.  According  a  phone  conversation  with  the  Operations/Planning 
Department  Head  for  NAVAIR  3.6.2,  the  LMDSS  system  has  always  been  funded  out  of 
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excess  year  end  funds  and  primarily  from  the  NALDA  program.  For  the  last  few  years, 
LMDSS  has  been  funded  "out  of  hide"  from  NAVAIR  3.6.2  Operations  and  Maintenance 
(O&M)  funds,  according  to  the  1999  LMDSS  Quality  Assurance  Leader.  This  constant 
scramble  to  obtain  funding  caused  numerous  hardships  during  the  entire  developmental 
process. 

In  general,  the  NALDA  AIS  codes  fall  under  Tab  G  of  the  Program  Objective 
Memorandum  (POM)  98/99  under  AIS  code  1274  (Naval  Air  Systems  Command, 
Acquisition  Program  Baseline,  1997).  The  costs  identified  with  the  parent  NALDA 
program  included  development,  implementation  and  operating  support  costs.  All  funds 
for  the  development  of  software  end  with  the  start  of  FYOO.  Implementation  funding  for 
software  is  minimal  until  FYOO,  when  it  triples,  but  it  should  be  noted  that  the  total 
amount  budgeted  once  the  implementation  starts  is  significantly  less  than  in  previous 
years  as  displayed  in  Table  1  below.  The  APB  also  stated  that  the  year-end  funds  are 
often  used  to  enhance  the  upgrade  and  modernization  schedules. 


IV  1999 

IT  1999 

IT  2000 

FY  2001 

sw 

Development 

3,758,000 

1,844,000 

0 

0 

SW 

Impletnentation 

250,000 

459,000 

681,000 

693,000 

Table  1  NALDA  Costs  excerpted from  NALDA  Phase  II  Acquisition  Program 

Baseline,  1997. 


At  this  time,  the  NALDA  program  has  missed  milestones  and  is  not  receiving  any 
additional  funding.  LMDSS  is  still  not  functioning  as  it  should  and  all  additional 
developmental  funding  is  being  redirected  to  getting  the  LMDSS  application  running 
smoothly.  This  is  hampering  the  developmental  efforts  of  other  systems  (Phone 
interview  Jahn/Evanoff,  1999). 
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5. 


Current  LMDSS  Scheduling  Plans 


Under  the  new  APB,  LMDSS  developers  now  had  to  meet  the  deadlines 
established  for  the  Milestone  m  Decision  Authority.  The  first  increment,  which 
consisted  of  the  consolidation  of  the  configuration  management  and  aviation  3M 
systems,  was  to  be  completed  by  February  1998.  Figure  2  displays  how  the  completion 
of  LMDSS  is  an  integral  part  of  the  NALDA  system  (NAVAIR  3.6,  1999).  In  the 
process  of  transitioning  from  one  software  language  and  platform  to  another  and  to  meet 
the  goals  of  the  NALDA  program,  many  DSS  items  were  discarded.  Graphics 
capabilities  and  some  of  the  statistical  forecasting  and  modeling  modules  were  scrapped. 

One  of  the  major  setbacks  is  the  lack  of  preformatted  reports  for  the  system. 
Nearly  50  reports  were  initially  developed  and  coded  but  since  the  structure  of  LMDSS 
changed  it  has  forced  the  continued  running  of  the  Mechanicsburg  site  in  order  to  ensure 
continuous  exchange  of  data  with  the  Fleet.  The  question  NAVAIR  is  now  trying  to 
answer  is  should  it  either  pay  to  update  the  original  coding,  or  take  a  different  attack  and 
develop  a  structured  queiy  language  based  means  of  obtaining  the  report  forms. 

Throughout  this  transition,  the  LMDSS  program  was  plagued  with  other  setbacks 
which  resulted  in  more  time  delays: 

•  A  Product  Support  Team  member  who  pushed  hard  to  transition  to  the 
HTML  format  was  transferred  taking  with  him  a  lot  of  corporate 
knowledge. 

•  The  merging  of  databases  resulted  in  some  corrupted  data. 

•  A  hacker  successfully  attacked  the  LMDSS  server  corrupting  invaluable 
data  and  resulting  in  LMDSS  being  taken  off  line  to  develop  additional 
security  measures  such  as  firewall  integration. 

•  Further  testing  revealed  LMDSS's  Active  X  programs  could  not  be 
downloaded  to  users  with  certain  web  browsers  who  were  not  accessing 
the  Internet  with  at  least  a  T-l  connection. 

•  The  resulting  decision  to  abandon  ActiveX  for  Java  caused  additional 
delays  due  to  training  deficiencies  and  data  reliability  problems. 
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In  1997,  proposed  performance  measures  were  written  for  NALDA  Phase  n. 
Areas  mentioned  in  this  document  related  to  the  LMDSS  system  were: 

•  User  satisfaction 

•  Logistic  support  costs 

•  Data  accuracy  and  timeliness 

Unfortunately,  this  document  did  not  accurately  outline  specific  performance 
measurements  that  could  be  used  in  the  developmental  or  implementation  phases  of  the 
LMDSS  project.  For  example,  the  User  Satisfaction  section  proposed  that  the 
effectiveness  of  the  act  of  combining  the  NALDA  II  subsystems  be  measured  through  the 
use  of  a  survey  conducted  six  months  after  the  technical  architecture  was  implemented. 
Although  a  functional  requirements  document  was  developed,  it  was  not  available  for 
review.  It  is  possible  that  more  specific  measures  were  developed  in  this  document. 
(Naval  Air  Systems  Command,  Performance  Measures,  Proposed,  1997) 
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IV.  METHODOLOGY 


A.  RESEARCH  QUESTIONS 

The  primary  research  question  is:  "Does  LMDSS  meet  user  requirements?"  The 
subsidiary  research  questions  mentioned  earlier  are: 

•  How  can  LMDSS  be  improved  to  better  meet  managerial  needs? 

•  What  are  the  significant  decisions  users  would  like  LMDSS  to  support? 

•  How  has  LMDSS  been  funded? 

•  How  can  NAVAIR  improve  the  LMDSS  program? 

•  What  can  future  software  developers  learn  from  studying  the  LMDSS 
project? 

•  What  are  the  key  attitudes  and  opinions  of  LMDSS  users? 


The  investigation  into  the  last  of  these  questions  led  to  the  additional  difference 
(group  comparison)  questions  below: 

•  Do  military  and  civilian  data  analysts  at  the  various  organizational  levels 
(fleet  command,  type  command,  system  command,  etc.)  have  significantly 
different  LMDSS  user  requirements? 

•  Do  various  categories  of  users  (officers,  enlisted.  General  Schedule~GS, 
civilian)  have  different  LMDSS  requirements? 

•  Is  there  a  difference  between  the  attitudes  of  LMDSS  users  at  various 
organizational  levels  concerning  modeling/simulation,  graphics,  exporting 
data,  and  the  ad-hoc  query  (IQ)  tool? 

•  Is  there  a  difference  in  the  frequency  of  LMDSS  use  among  the  LMDSS 
organizational  groupings? 
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B.  DESIGN  OF  THE  STUDY 


This  research  is  best  described  as  an  ex-post  facto  design  consisting  of  a  one-shot 
questionnaire  survey  (Campbell  &  Stanley,  1963)  triangulated  with  face-to-face  and 
phone  interviews.  The  term  ex-post  facto  is  an  appropriate  description  because  the 
research  team  did  not  control  or  manipulate  the  independent  and  dependent  variables 
identified.  The  research  conducted  on  the  LMDSS  project  was  exploratory  or  pre- 
experimental  in  nature.  Simple  random  samples  were  chosen  from  a  population  of  843 
LMDSS  users  registered  with  NAVAIR  3.6.2.  A  Web-based  survey,  face-to-face 
interviews,  and  phone  interviews  were  the  methods  used  to  collect  data.  Past  NAVAIR 
LMDSS  briefs  and  documents  posted  on  the  NAVAIR  3.6.2  Web  site  were  secondary 
sources  of  data  (NAVAIR,  May  1999). 

C.  CONDUCT  OF  THE  STUDY 
1.  Research  Strategy 

a)  Prestudy 

The  prestudy  phase  of  this  research  began  in  November,  1998  during  a 
three-day  visit  to  NAVAIR  3.6.2  at  the  NAS  Patuxent  River,  Maryland.  Fifteen  personal 
interviews  were  conducted  with  an  emphasis  on  questioning  those  individuals  directly 
involved  in  the  management  of  the  LMDSS  project.  Most  of  the  interviews  were 
recorded  and  notes  were  used  to  document  the  minor  discussions  that  took  place.  An 
exploratory  focus  group  was  held  on  the  final  day  of  the  visit  with  the  three  individuals 
assigned  to  the  LMDSS  Help  Desk.  The  purpose  of  the  focus  group  was  to  gain 
additional  insight  missed  during  the  interviews  and  help  develop  meaningful  questions 
for  the  forthcoming  survey.  The  focus  group  was  moderated  by  the  thesis  authors  and 
videotaped.  See  Appendix  C  for  a  complete  list  of  the  questions  that  were  asked.  An 
Access  97  database  containing  the  total  population  of  843  LMDSS  users  was  received 
from  NAVAIR  during  this  visit. 
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Upon  leaving  NAVAIR,  the  research  team  traveled  to  NAS  Norfolk  and 
NAS  Oceana  completing  five  personal  interviews  with  LMDSS  users  at  the  Safety 
Center,  Aviation  Intermediate  Maintenance  Depot  (AIMD),  Wing,  and  Squadron  levels. 
These  interviews  were  exploratory  in  nature  and  were  documented  in  the  same  manner  as 
those  conducted  at  NAS  Patuxent  River. 

b)  The  Pilot  Study 

The  pilot  study  phase  consisted  of  choosing  two  random  samples  from  the 
LMDSS  user  population,  developing  and  pilot  testing  phone  interview  and  survey 
questions,  validating  email  addresses  for  users  within  the  samples,  mastering  the  web 
survey  software,  and  setting  up  a  web  site  to  host  the  survey. 

Two  systematic,  random  samples  (Cooper  and  Emory,  1995)  were 
selected  from  the  total  population  of  843  LMDSS  users.  A  random  sample  of  289 
LMDSS  users  was  selected  to  participate  in  the  Web-based  survey.  A  smaller  sample  of 
twenty-five  LMDSS  users  was  chosen  to  participate  in  the  phone  interviews. 

The  phone  interview  and  survey  questions  were  developed  from  the 
information  collected  during  the  personal  interviews  and  focus  group  conducted  in  the 
prestudy  phase.  The  initial  drafts  of  both  the  phone  interview  questions  and  the  survey 
questions  were  emailed  to  the  twenty-eight  member  LMDSS  Quality  Assurance  Team  for 
feedback.  The  QA  team  consisted  of  logistics  and  aviation  maintenance  experts  within 
the  NAVAIR  domain  who  were  selected  by  the  LMDSS  program  manager  to  assist  in 
testing  LMDSS  modules.  The  final  drafts  of  both  the  phone  interview  and  survey 
questions  were  developed  from  the  email  critiques  and  phone  comments  received  from 
the  QA  group. 

The  most  time  consuming  task  in  the  pilot  study  phase  was  validating  the 
email  addresses  of  the  289  and  25  member  random  samples.  The  LMDSS  database 
received  from  NAVAIR  had  an  abundance  of  entries  dating  back  from  two  to  four  years. 
Of  the  original  289  LMDSS  users  selected  to  participate  in  the  Web  survey,  only  213 
could  be  contacted  by  phone  to  attain  their  correct  email  addresses.  All  of  the  213  users 
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contacted  had  access  to  the  Web  and  were  able  to  provide  valid  email  addresses.  These 
213  users  became  the  effective  random  sample  for  the  Web  survey. 

The  software  application  chosen  to  implement  the  web-based  survey  was 
Perseus  Survey  Solutions  for  the  Web.  Once  the  completed  survey  questions  were 
entered  into  Perseus,  they  were  transformed  into  a  HTML  format  and  posted  on  the  Web. 
Coding  within  the  survey  was  designed  to  send  survey  responses  to  a  central  Perseus 
server  where  they  were  encapsulated  and  sent  back  as  email  to  the  research  team. 

c)  The  Study 

The  study  phase  involved  conducting  the  phone  interviews,  soliciting  Web 
survey  responses  from  sample  members,  documenting  the  phone  interviews,  collecting 
the  survey  responses,  and  sending  out  individual  thank-you  emails. 

Phone  interviews,  each  lasting  thirty  to  forty-five  minutes,  were  conducted 
over  a  period  of  four  months.  Seventeen  of  the  twenty-five  member  random  sample  were 
interviewed  by  phone  as  well  as  various  LMDSS  managers  at  NAVAIR  3.6.2.  As  the 
research  continued,  additional  questions  arose  that  were  not  asked  on  the  initial  NAVAIR 
visit.  Volunteer  graduate  students  from  the  Naval  Postgraduate  School,  who  participated 
in  related  group  projects,  were  tasked  to  contact  members  of  the  NAVAIR  3.6.2  staff  for 
updated  LMDSS  information.  The  authors  of  this  thesis  developed  the  questions  that 
were  used  and  the  answers  received  have  been  incorporated  throughout  this  body  of 
work. 

Initial  emails  requesting  survey  participation  were  simultaneously  sent,  or 
“shotgunned”,  to  all  213  members  of  the  effective  random  sample.  As  users  replied,  they 
were  dropped  from  the  shotgun  list  and  sent  thank-you  emails.  Email  addresses  were 
constantly  changing  throughout  the  study  phase  of  the  research.  Even  after  all  213  emails 
had  been  verified,  at  least  ten  to  fifteen  emails  had  to  be  corrected  after  each  of  four 
attempts  to  solicit  additional  survey  responses  from  sample  members.  After  the  third 
solicitation  attempt,  approximately  120  of  the  213  LMDSS  users  contacted  replied.  A 
final  round  of  ninety-three  individualized  emails  were  sent  to  the  remaining  non- 
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participants  requesting  assistance  in  completing  the  survey.  Thirty-two  additional 
responses  were  received.  After  four  weeks,  a  total  of  152  LMDSS  users  from  the  213- 
member  sample  replied  to  the  Web  survey.  Individual  thank-you  emails  were  sent  to 
each  replying  sample  member  within  one  day  of  receiving  a  response. 

Phone  interviews  were  taped,  with  the  permission  of  the  interviewees,  and 
later  transcribed  into  notes.  Only  the  interview  questions  developed  during  the  prestudy 
and  tested  during  the  pilot  study  were  used  during  the  phone  interviews  with  individuals 
from  the  twenty-five  member  sample.  Additional  sets  of  questions  were  customized  for 
the  each  of  the  NAVAIR  LMDSS  managers  contacted  by  graduate  students  throughout 
the  study  phase.  Phone  interviews  proved  difficult  to  complete  because  interview 
appointments  were  often  cancelled  at  the  last  minute  due  to  interviewee  work  conflicts. 
After  months  of  interviewing  it  became  more  efficient  to  conduct  the  interview  during 
the  initial  contact  phone  call  instead  of  making  future  appointments. 

D.  THE  SAMPLE 

1.  Random  Selection  Procedure 

The  purpose  of  random  sampling  in  this  study  was  to  attain  samples  of  users  that 
represented  the  characteristics  of  the  larger  population  of  LMDSS  users.  It  can  be  argued 
that  studying  these  samples  enables  accurate  conclusions  to  be  reached  about  the  larger 
LMDSS  user  population.  In  order  to  strengthen  this  argument,  a  systematic  random 
selection  process  was  implemented  to  capture  at  least  fifteen  percent  of  the  population  to 
participate  in  the  Web  survey.  Planning  for  a  maximum  non-participation  rate  of  fifty 
percent,  the  initial  size  of  the  survey  sample  was  approximately  twenty  five  percent  of 
the  population.  The  systematic  random  selection  process  began  when  a  seed  number 
between  one  and  eight  was  literally  drawn  out  of  a  hat  to  select  the  first  member  of  the 
sample.  The  seed  number  drawn  for  the  larger  of  the  two  samples  was  a  five.  Every 
fourth  user  was  then  selected  from  the  843  LMDSS  user  database  which  was 
alphabetically  ordered  by  last  name.  The  resulting  sample  consisted  of  289  users.  A 


41 


similar  procedure  was  chosen  to  select  the  smaller,  second  random  sample  of  twenty-five 
names.  The  seed  number  randomly  chosen  was  seven.  The  289  user  random  sample  was 
chosen  to  participate  in  the  Web  survey  while  the  twenty-five  user  random  sample  was 
chosen  to  participate  in  the  phone  interviews.  Additional  Access  97  databases  were 
developed  to  manage  both  random  samples.  (Cooper  and  Emory,  1995) 

It  was  assumed  that  the  size  of  the  twenty-five  user  sample  chosen  for  the  phone 
interviews  was  too  small  to  enable  precise  estimations  or  inferences  about  population 
characteristics.  The  primary  purpose  of  creating  the  smaller  sample  was  to  attempt  to 
gain  some  additional  insight  by  using  open-ended  questions  that  were  more  appropriately 
delivered  by  a  phone  interview  than  a  survey.  Time  and  workload  limitations  also 
influenced  the  decision  to  limit  the  number  of  random  phone  interviews. 

2.  Validation  of  Email  Addresses 

The  NAVAIR  LMDSS  user  database  contained  many  old  and  outdated  entries. 
This  called  for  an  aggressive  effort  to  clean  the  data.  Cleaning  the  user  contact 
information  involved  validating  the  phone  number,  email,  identity,  and 
command/organization  of  over  350  users  contained  in  the  two  random  samples,  the  QA 
team,  and  managers  at  NAVAIR.  During  the  period  of  this  research,  many  commands 
transitioned  to  Information  Technology  for  the  Twenty  First  Century  (IT-21)  standards. 
This  transition  resulted  in  significant  changes  to  email  addresses  throughout  the  Navy 
during  the  study  phase  of  this  research.  Maintaining  current  email  addresses  on  the  users 
belonging  to  the  two  samples  required  an  almost  daily  effort  and  proved  much  more  time 
consuming  than  anticipated.  Two  to  three  phone  calls  were  often  required  to  contact 
each  sample  member.  Many  email  addresses  were  retrieved  only  to  change  a  few  months 
later  as  Navy  commands  updated  their  networks  to  meet  IT-21  standards.  Table  2  shows 
the  breakout  of  how  the  original  random  sample  of  289  names  chosen  for  the  web-based 
survey  decreased  to  213  names  as  a  result  of  the  phone  number/email  validation.  Three 
members  who  were  contacted  declined  to  participate  because  they  believed  their  contact 
with  the  program  was  so  minimal  that  they  could  not  provide  any  relevant  data. 
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Of  the  twenty-five  LMDSS  users  selected  to  participate  in  the  phone  interviews, 
only  seventeen  were  contacted.  Researchers  were  unable  to  reach  six  of  the  sample 
members  due  to  changes  in  both  phone  numbers  and  email  addresses.  The  remaining 
two  users  failed  to  keep  previously  arranged  appointments  on  several  occasions  and  did 
not  return  subsequent  phone  messages. 


Reasons  for  Sample  Shrinkage 

Number  of  LMDSS  users 

Original  Random  Sample 

289 

Declined  to  Participate 

(3) 

Transferred 

(31) 

Retired 

(5) 

Changed  to  an  unknown  Phone  #  and  Email  address 

(32) 

Effective  Sample  Remaining  for  Web  Survey 

213 

Total  Number  of  LMDSS  Users  Who  Responded 

152 

Table  2  Shrinkage  of  Original  Random  Sample  (Web-based  Survey) 


3.  Testing  the  Effective  Sample  for  Randomness 

Due  to  the  large  attrition  rate  of  the  original  289-user  sample,  randomness  could 
no  longer  be  assumed.  Additional  testing  was  completed  to  ensure  that  the  effective 
sample  of  213  LMDSS  users  was  still  random  in  nature.  Using  the  same  systematic, 
random  selection  procedure  described  earlier  (the  random  seed  number  selected  was 
eight)  a  random  test  sample  of  105  LMDSS  users  was  created  to  be  compared  with  the 
effective  sample  of  213  users. 

The  demographic  characteristics  of  both  independent  samples  were  measured 
based  on  data  found  within  the  NAVAIR  database.  For  each  sample,  users  were  assigned 
to  mutually  exclusive  and  exhaustive  user  categories  defined  as:  Officer,  Enlisted, 
General  Schedule  (GS),  Civilian,  and  Unknown.  All  members  of  both  samples  were  then 
assigned  to  mutually  exclusive  and  exhaustive  organization  types  consisting  of:  Fleet 
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Command,  System  Command,  Type  Command,  Program  Office  (PMA/APML),  Depot, 
AJMD,  WING,  Squadron,  Marine,  Reserve,  Private  Company,  and  Unknown.  The 
demographic  data  for  both  samples  is  depicted  in  Tables  3  and  4. 

The  chi-square  nonparametric  test  was  chosen  to  compare  the  two  independent 
sample  groups  because  the  demographic  data  as  shown  in  the  above  tables  was  nominal 
and  categorical.  The  requirements  of  independence  between  the  groups,  and  mutually 
exclusive/exhaustive  categorical  data  were  met. 


Officer 

Enlisted 

os 

Civilian 

In  known 

Effective 
Sample  of 
213  users 

26 

48 

84 

47 

8 

Test 

Sample  of 
105  users 

16 

31 

32 

22 

4 

Table  3  Sample  Demographics  With  Respect  to  User  Category 


Organization  Type 

Effective  Sample  of  213 
Users 

Test  Sample  of  105 
Isers 

Fleet  Command 

7 

5 

System  Command 

43 

14 

Type  Command 

5 

4 

12 

3 

Depot 

27 

11 

AJMD 

8 

7 

Wing 

9 

6 

Squadron 

5 

6 

Marines 

15 

8 

Reserves 

6 

1 

Private  Company 

46 

46 

Unknown 

30 

30 

Table  4  Sample  Demographics  With  Respect  to  Organization  Type 
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The  null  hypothesis  notation  is:  H  0  :  Ei  =  Tr ,  where  E  is  the  effective  random 
sample  and  T  is  the  test  sample.  The  null  hypothesis  claims  there  is  no  difference 
between  the  demographic  characteristics  of  the  effective  213-user  sample  and  the  105- 
member  test  random  sample.  The  level  of  significance  (the  probability  of  rejecting  a  null 
hypothesis  that  is  true)  is  .05.  Critical  values  of  the  chi-square  distribution,  based  on 
degrees  of  freedom,  were  found  in  a  statistical  table  from  Cooper  and  Emory  (1995,  p. 
659).  The  chi-square  tests  comparing  the  samples,  using  SPSS  statistical  software, 
resulted  in  the  output  displayed  in  Table  5. 


lest 

C  ritical  Value  of 

Degrees  of 

SPSS  Calculated 

(.05  level  of 

Chi-Square 

Freedom 

Critical  Value  of 

significance) 

Distribution 

Chi-Square 

User  Category 
Comparison 
Between  Samples 

9.49 

4 

3.461 

Organizational 
Type  Comparison 
Between  Samples 

19.68 

11 

9.706 

Table  5  Chi-Square  Tests  of  the  Two  Independent  LMDSS  Samples 


In  both  chi-square  tests  comparing  the  user  categories  and  organization  types  of 
the  two  samples,  the  SPSS  calculated  critical  value  was  less  than  the  critical  value  of  the 
chi-square  distribution.  The  null  hypothesis  therefore  cannot  be  rejected.  The  samples 
are  not  significantly  different  at  a  significance  level  of  .05.  Since  the  effective  sample 
and  test  sample  were  not  significantly  different,  the  argument  can  be  made  that  the 
effective  sample  of  213  LMDSS  users  is  indeed  random  and  representative  of  the  greater 
LMDSS  user  population. 

E.  INSTRUMENTATION 

1.  Need  for  a  Custom  LMDSS  Survey 

Since  LMDSS  is  a  custom  software  application  developed  by  NAVAIR  for  the 
Navy's  use,  there  was  not  a  commercial,  off-the-shelf  (COTS)  survey  instrument 
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available  considered  appropriate  for  the  requirements  of  this  research.  The  questionnaire 
developed  by  Moore  and  Snyder  (1998)  was  designed  for  a  different  audience  with  a 
different  research  focus  thus  their  instrument  did  not  meet  the  needs  of  this  research 
effort.  COTS  software  was  used  to  implement  the  Web  survey  (Perseus),  manage  the 
user  samples  (Access  97),  and  statistically  analyze  our  survey  results  (SPSS  v.9.0  and  S- 
Plus  v.  4.0). 

2.  Developing  and  Pilot  Testing  the  Survey  Instrument 

In  order  to  collect  as  many  responses  as  possible  from  the  213  member  effective 
random  sample,  it  was  decided  to  develop  a  Web-based  survey.  It  was  thought  the 
novelty  of  a  paperless  Web  survey  would  attract  more  responses  and  limit  non-response 
error.  Email  was  used  to  conduct  multiple,  rapid  requests  for  user  participation. 
Individually  addressed  emails  seemed  to  generate  a  significantly  greater  response  rate 
than  group  or  shotgun  emails  to  multiple  users. 

The  initial  draft  of  the  survey  questions  was  created  from  the  interviews  and  a 
focus  group  conducted  at  NAVA®.  The  focus  group  was  responsible  for  many  of  the 
questions  relating  to  the  quality  of  LMDSS  training,  the  IQ  tool,  and  how  LMDSS  is 
used.  Pretesting  was  accomplished  by  conducting  a  pilot  test  of  the  Web  survey  to  detect 
any  design  weaknesses  in  the  survey  instrument.  The  pilot  survey  was  posted  on  the  Web 
and  the  twenty-eight  member  LMDSS  QA  team  was  contacted  by  email  and  requested  to 
provide  feedback.  Thirteen  survey  responses  were  received.  As  a  result  of  these 
responses,  the  number  of  open-ended  questions  in  the  survey  were  reduced,  several 
questions  considered  confusing  were  rewritten,  and  options  such  as:  not  applicable, 
unknown^  and  don't  know  were  included.  The  final  version  of  the  Web  survey  is  found  in 
Appendix  D. 

3.  Pilot  Testing  the  Phone  Interview  Questions 

The  phone  interview  questions  were  developed  in  a  similar  manner  as  the  survey. 
Initial  draft  questions  were  created  using  input  from  interviews  and  the  focus  group 
conducted  at  NAVA®.  The  pilot  questions  were  emailed  to  the  LMDSS  QA  team  for 
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evaluation.  As  a  result  of  the  QA  team's  feedback,  several  questions  were  added  and 
several  were  rewritten  for  better  clarity.  The  majority  of  phone  interview  questions  were 
open-ended,  to  allow  a  level  of  probing  and  depth  not  considered  possible  by  the  Web 
survey  alone.  Fifteen  of  the  seventeen  phone  interviews  conducted  were  done  by  the 
same  researcher  to  minimize  interviewer  bias.  The  questions  developed  for  the  random 
phone  interviews  are  presented  in  Appendix  E. 

F.  INDEPENDENT  AND  DEPENDENT  VARIABLES 

Many  authors  restrict  the  term  independent  variable  (IV)  to  variables  that  are 
manipulated.  Some  researchers  define  an  independent  variable  more  broadly  to  include 
any  predictors,  presumed  causes,  or  influences  under  investigation.  Such  independent 
variables  that  define  attributes  of  persons  or  the  environment  and  are  not  manipulated  in 
the  study  are  called  attribute  independent  variables.  (Morgan  and  Griego,  1998) 

The  attribute  IV  chosen  as  a  possible  influence  on  the  dependent  variables 
measured  in  the  Web  survey  was  the  organization  type  as  previously  defined  in  the  chi- 
square  testing  of  the  sample  members.  Cooper  and  Emory  (1995)  define  an  extraneous  or 
moderating  variable  (MV)  as  a  second  IV  that  is  considered  because  it  is  believed  to  have 
an  influence  on  the  original  independent/dependent  variable  relationship.  User  category 
(officer,  enlisted.  General  Schedule  (GS),  civilian)  was  suspected  to  be  a  MV.  Morgan 
and  Griego  (1998)  describe  a  dependent  variable  (DV)  as  the  presumed  outcome  that 
measures  the  effect  of  the  independent  variable.  An  IV  typically  affects  a  DV  in  a  way 
that  can  be  measured.  Surveys  and  interviews  are  two  ways  to  measure  such  an  effect. 

The  DVs  that  were  measured  in  the  Web  survey  for  possible  IV  effects  to  answer 
the  difference  based  research  questions  were: 

•  Importance  of  modeling  and  simulation 

•  Importance  of  a  graphics  capability 

•  Importance  of  a  capability  to  cleanly  export  LMDSS  data 

•  Importance  of  a  structured,  ad-hoc  query  (IQ)  tool 
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Frequency  of  LMDSS  use. 


G.  ANALYSIS  STRATEGY 

Research  conducted  with  attribute  variables  is  limited  in  its  ability  to  establish 
proof  of  causality  with  any  certainty.  The  standard  of  causation  requires  more  complex, 
experimental  research  approaches  where  active,  independent  variables  are  manipulated 
over  time.  The  scope  of  this  research  is  limited  to  answering  the  main  research  questions, 
and  exploring  the  aforementioned  differences  between  the  LMDSS  user  groups. 

Data  from  the  phone  interviews  were  transcribed  from  tape  recordings  made  by 
the  interviewers.  The  interview  transcripts  were  analyzed  to  identify  common  user 
attitudes  and  perceptions.  Trends  were  uncovered  and  compared  with  those  found  in  the 
Web  survey  responses. 

S-Plus  statistical  software  was  used  to  conduct  a  frequency  analysis  on  the  closed- 
response  (specified  alternatives)  questions  within  the  Web  survey.  Content  analysis  was 
completed  on  the  responses  provided  from  the  open-ended  survey  questions.  One-way 
Analysis  of  Variance  (ANOVA)  testing  matching  one  of  two  possible  IVs  (organizational 
type  and  user  category)  with  each  DV  measured  in  the  survey  was  conducted  using  SPSS 
statistical  software.  The  one-way  ANOVA  was  used  to  assist  in  answering  the  difference 
(group  comparison)  research  questions. 

Open-responses,  free  choice  of  words,  from  the  Web  survey  were  collected  and 
compared  using  content  analysis  with  the  other  types  of  data  to  provide  answers  to 
specific  research  questions.  Data  from  phone  interviews  with  managers  at  NAVAIR 
3.6.2  was  reviewed,  summarized,  and  compared  with  other  data  sources. 
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V.  FINDINGS 


A.  ORGANIZATION  OF  FINDINGS 

The  findings  in  this  chapter  are  presented  in  the  following  sections: 

•  Frequency  and  content  analysis 

•  Establishing  difference  null  hypotheses 

•  ANOVA  tests 

•  Phone  interview  findings 

•  Interviews  with  past  and  present  LMDSS  managers 

1.  Frequency  and  Content  Analysis 

The  material  in  the  first  section,  frequency  and  content  analysis,  includes  a 
detailed  analysis  of  the  data  collected  from  the  Web-based  survey.  To  assist  in  analysis, 
the  survey  questions  were  grouped  into  the  following  subheadings  according  to  the  topic 
covered  in  each  question: 

•  Demographics 

•  Frequency  of  use 

•  Hardware/software  concerns 

•  Usage  of  application 

•  Modeling/simulation 

•  Training 

•  Graphics 

•  User  perceptions. 
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The  presentation  of  the  Web  survey  responses  mirrors  the  approximate  order  of 
the  questions  as  found  in  the  web  based  survey  questionnaire  (See  Appendix  D). 

A  variety  of  statistical  software  applications  were  used  to  conduct  the  analysis. 
SPSS  version  9.0,  S-Plus  version  4.0  and  Excel  97  were  used  to  generate  the  charts  and 
graphs  contained  in  this  chapter.  A  combination  of  summary  tables,  percentage 
histograms  and  figures  were  used  to  accurately  display  the  findings  for  each  survey 
question. 

2.  Establishing  Difference  Null  Hypotheses 

In  the  second  section,  establishing  difference  null  hypotheses,  the  null  hypotheses 
explored  in  this  thesis  are  established  based  on  the  difference  questions  previously 
identified  in  the  methodology  chapter.  These  null  hypotheses  were  developed  for  the 
difference  questions  derived  from  the  last  research  question  only.  These  hypotheses  were 
tested  via  a  one-way  ANOVA  strategy.  The  remaining  research  questions  were  explored 
using  frequency  and  content  analysis  of  the  various  data  sources  as  previously  discussed. 

3.  ANOVA  Tests 

In  the  third  section,  ANOVA  tests,  the  parametric  statistic  known  as  one-way 
ANOVA  was  used  to  test  the  difference  null  hypotheses  to  assist  in  answering  the  final 
group  comparison  research  questions. 

4.  Phone  Interviews 

The  fourth  section,  phone  interviews,  presents  a  summary  of  the  data  obtained 
through  seventeen  phone  interview  surveys.  This  data  collection  tool  was  used  to  collect 
data  not  easily  captured  through  the  web  survey. 

5.  Interviews  with  Past  and  Present  Managers 

To  get  a  historical,  managerial  perspective  phone  and  face-to-face  interviews 
were  conducted  during  the  course  of  this  research  project  with  past  and  present  LMDSS 
managers.  A  summary  of  these  findings  is  contained  in  the  final  section  of  this  chapter. 
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B.  FREQUENCY  AND  CONTENT  ANALYSIS 


1.  Demographics 

The  two  IVs  considered  in  this  thesis  are  the  user  categories  and  the  various 
organization  types.  User  category  demographic  data  will  be  referred  to  as  IV- 1  and  the 
organization  type  as  IV-2. 

The  user  category  groups  (IV-1)  were  broken  into  four  subsets:  Officer,  Enlisted, 
General  Service  (GS)  and  Civilian.  IV-1  demographic  data  was  obtained  through  the  use 
of  the  original  NAVAIR  LMDSS  user  database.  A  result  of  the  breakdown  of  the 
respondents  is  displayed  below  in  Table  6. 


Group  Name 

N  umber  of  Responses 

Percentage  of  152 
Respondents 

Officer 

14 

9% 

Enlisted 

25 

16% 

General  Service 

74 

49% 

Civilian 

39 

26% 

Total 

152 

100% 

Table  6  User  Category  Groups  (IV-1)  Summary  Results 


The  organization  type  was  obtained  through  direct  input  by  the  survey 
respondents.  The  resulting  organizational  categories  are  listed  in  Table  7.  The  data 
found  in  Table  7  shows  a  majority  of  the  responses  came  from  survey  members  attached 
to  system  commands,  civilian  firms  under  contract  to  the  Navy,  and  depots.  A  total  of  65 
percent  of  the  total  survey  response  data  was  received  from  personnel  in  these  three 
organization  types.  The  two  organizations,  whose  members  did  not  reply  to  the  survey, 
Commander  Naval  Reserve  Force  (COMNAVRESFOR)  and  the  Naval  Safety  Center, 
were  automatically  deleted  from  this  and  all  following  histograms  to  facilitate  easier 
analysis.  A  reason  for  the  non-response  from  users  within  these  two  commands  is  not 
known. 
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Organization  Type 

Number  of 
Responses 

Percentage  of  152 
Respondents 

Systems  Command 
(COMNAVAIRSYSCOM, 

SPAWAR...) 

40 

26% 

Civilian  Firm  under  contract  to  the  Navy 

36 

24% 

Depot 

23 

15% 

Other 

14 

9% 

Wing 

9 

6% 

Squadron 

7 

5% 

Aircraft  Intermediate  Maintenance 

Dept. 

6 

4% 

Type  Command  (COMNAVAIRPAC, 
COMNAVAIRLANT...) 

5 

3% 

Marine  Corps  Activity 

5 

3% 

Fleet  Command  (CINCLANT, 

CINCPAC,  COMTHIRDFLT. . . ) 

3 

2% 

COMNAVAIRRESFOR  ' 

2 

1%  ^ 

Navy  Inventory  Control  Point 

2 

1  % 

Naval  Safety  Center 

0 

0% 

COMNAVRESFOR 

0 

0% 

Total 

i _ 152 _ 

100  % 

Table  7  Organization  Type  Groups  (IV-2)  Summary  Results 


2.  Frequency  of  Use 

Survey  questions  1,  3  and  4  as  found  in  Appendix  D  are  covered  in  this  section. 
For  ease  of  use,  each  figure  in  this  chapter  containing  a  percentage  histogram  also 
displays  the  specific  question  as  asked  in  the  original  survey. 

Figure  3  displays  the  percent  histogram  for  question  1,  “How  often  did  you  access 
LMDSS  when  it  was  available  on  the  web?”  It  shows  less  than  15  percent  of  all 
responding  users  accessed  the  LMDSS  program  on  a  daily  or  weekly  basis.  Over  41 
percent  used  the  application  infrequently  and  26  percent  answered  Not  Applicable.  This 
may  partially  be  attributed  to  the  long  period  of  time  that  the  program  was  frozen.  New 
users  were  still  being  issued  passwords  but  were  unable  to  access  the  system. 
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Figure  3  Frequency  of  Usage  Percent  Histogram 
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1  -  2  years  3  -  4  years  Dont  remember  More  than  5  years 

2- 3  years  6 -12  months  Less  than  6  months  Not  appicabte 

Q3  How  long  Iras  It  been  since  you  lost  accessed  LMDSS? 


Figure  4  Duration  Since  Last  Access  Percent  Histogram 
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Q4  When  were  you  first  introduced  to  LMDSS? 


Figure  5  Length  of  Time  Using  LDSS  Percent  Histogram 

Questions  3  and  4  are  directly  related  to  each  other  and  are  covered  in  Figures  4 
and  5.  Question  3  asked,  “How  long  it  has  been  since  you  last  accessed  LMDSS?”  and 
question  4  queried,  “When  were  you  first  introduced  to  LMDSS?”  From  Figure  5,  it  is 
apparent  that  over  half  of  the  users  signed  onto  the  system  in  the  last  two  years  yet  only  6 
percent  of  respondents  have  signed  on  in  the  last  six  months. 

3.  Hardware/Software  Concerns 

Results  of  questions  5  through  8  are  discussed  in  this  section.  Figure  6  indicates 
the  findings  of  question  5,  “When  the  LMDSS  Web  site  comes  back  online,  what  will  be 
your  primary  means  of  accessing  it?”  As  shown  by  the  percent  histogram,  nearly  60 
percent  of  all  respondents  reported  they  accessed  LMDSS  through  the  network. 
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QSVAwnthB  LMDSS  site  comes  back  on  nine,  what  wi  be  your  primary  means  of  tccesing  It? 


Figure  6  Primary  Means  of  Access  to  LMDSS  Percent  Histogram 

Question  6  asked  the  respondent  to  check  all  answers  that  applied  to  the 
following  question:  “What  kind  of  problems  did  you  have  accessing  LMDSS?”  Results 
are  tabulated  in  Table  8. 


Answer 

Number  of 
Responses 

Percentage  of  152 
Respondents 

1  have  not  had  any  problems 

51 

33% 

Don’t  know,  just  could  not  access  the 
LMDSS  web  site 

32 

21% 

Other 

32 

21  % 

Firewall  blocking  access 

23 

15% 

Downloading  matrix  tables 

19 

12% 

Unable  to  access  the  IQ  tool 

11 

7% 

Computer  hardware  inadequate 

10 

7% 

Internet  browser  issues 

8 

5% 

Proxy  difficulties 

6 

4% 

Total  without  difficulties: 

51 

34  % 

Total  with  one  or  more  difficulties 

101 

66 % 

Total  with  two  difficulties 

17 

11% 

Total  with  three  or  more  difficulties 

11 

7% 

Table  8  Type  of  Access  Difficulties  Table  Summary 
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It  should  be  noted  that  although  101  people  reported  at  least  one  difficulty,  only 
twenty-eight  reported  encountering  more  than  one  difficulty.  On  the  other  hand,  forty- 
two  percent  of  the  respondents  responded  with  “don’t  know”  or  had  some  “other”  type  of 
problem.  Without  knowing  the  experience  level  of  the  users  and  the  specifics  of  their 
inability  to  access  the  site,  it  is  impossible  to  determine  exactly  what  type  of  difficulty 
these  respondents  encountered.  They  may  have  encountered  unique  problems  or  they 
may  simply  have  been  unable  to  identify  the  problem. 

Table  9  presents  the  cross  tabulation  summary  statistics  between  IV-2 
(organization  type)  and  the  type  of  difficulties  encountered  by  the  surveyed  users.  It  is 
interesting  to  note  that  civilian  firms  appeared  to  encounter  firewall  difficulties  at  a 
greater  rate  than  their  government  counterparts.  Twenty-five  percent  of  all  civilian 
contractors  surveyed  and  nearly  twenty-two  percent  of  all  depot  respondents  reported 
firewall  trouble.  Systems  commands,  which  were  the  largest  of  the  surveyed 
organizations,  also  reported  firewall  difficulties  but  at  a  much  lower  rate  of  7  percent. 


Type  of  Difficulty 
and  Total  Number 

Top  Three  Organizations  and 

Number  of  Difficulties  per  Organization 

Firewall 

23 

Civilian  Firm 

9 

Depot 

5 

SYSCOM 

TYCOM 

3 

Downloading 
matrix  tables 

19 

Civilian  Firm 

.  .  . 

6 

Depot 

5 

Wing 

3 

Unable  to  access 
the  IQ  tool 

11 

Depot 

H 

Civilian  Firm 

3 

Wing 

3 

Computer  HW 
inadequate 

10 

Depot 

H 

AIMD 

2 

USMC 

1 

Internet  browser 

8 

USMC 

2 

Civilian  Firm 

2 

NA 

Proxy 

6 

Civilian  Firm 

3 

Depot 

TYCOM 

1 

Other 

32 

SYSCOM 

11 

Civilian  Firm 

7 

USMC 

Other 

Depot 

3 

Don’t  know 

32 

Civilian  Firm 
SYSCOM 

9 

Depot 

Other 

5 

NA 

Table  9  Summary  of  Access  Difficulties  by  Type  Organization 
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In  addition  to  the  firewall  access  problem,  a  lot  of  users  answered  with  “other”  or 
“don’t  know”.  Not  surprisingly,  the  three  organizations  that  comprise  the  largest 
percentage  of  the  total  survey  respondents  (systems  commands,  civilian  firms  and  depots) 
reported  the  largest  number  of  “other”  and  “don’t  know”  responses. 

Although  the  systems  commands  provided  the  greatest  number  of  respondents, 
many  of  the  more  specific  access  problems  were  identified  by  the  depot  level  users  who 
lodged  a  greater  quantity  of  complaints.  Part  of  this  might  be  attributed  to  the  fact  that 
depot  users  provided  the  largest  number  of  complaints  concerning  inadequate  computer 
hardware.  There  may  be  a  correlation  between  inadequate  hardware  and  the  access 
problem  but  further  study  would  be  required. 

Question  7  asked,  “When  LMDSS  is  back  online,  users  will  be  required  to  use 
Netscape  Communicator  4.5  with  128  bit  encryption.  Do  you  anticipate  any  problems 
downloading  and  installing  this  browser?”  Figure  7  presents  the  results  of  this  question. 

60 

40 

Oft 

n 

i 

at 

a 

£ 

20 

0 

Dent  know  No  Not  appicoNe  Yes 

Q?  When  LMDSS  is  back  online,  users  will  be  required  to  use  Netscape  Communicator  4.5 

with  128  bit  encryption  (available  for  free  download  at  wwwjnetscapexom).  Do  you  anticipate 

any  problems  downloading  and  InstaSng  this  browser? 

Figure  7  Browser  Installation  Percent  Histogram 

As  shown  in  this  question  result,  there  does  not  seem  to  be  any  anticipated 
difficulty  in  the  change  of  browsers. 
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Question  8  was  an  open-ended  question  that  queried  users  on  “What  type  of 
software  problems  did  you  experience  while  using  LMDSS?”  Table  10  summarizes  the 
results  of  this  query. 


Category 

Number  of  Responses 

Percentage  of  152 
Respondents 

Not  Applicable 

95 

63% 

No  Problems 

30 

20% 

Entered  comment 

27* 

18  % 

Never  used  LMDSS 

5 

3% 

Left  completely  blank 

2 

1% 

*  Seven  users  who  reported  no  problems  also  entered  comments. 

Table  10  Summary  of  Respondents  Reporting  Software  Problems 


Although  only  17.8  percent  of  all  respondents  entered  comments  about  the 
problems  they  experienced  while  using  the  LMDSS  application,  many  of  those  who  did 
reply  entered  multiple  areas  of  concern.  Several  themes  of  dissatisfaction  were  easily 
identified  because  they  were  repeated  throughout  the  commentary.  The  top  five  concerns 
raised  in  the  dialog  from  the  question  8  submissions  were  as  follows: 

•  Inability  to  access  LMDSS 

•  LMDSS  was  not  easy  to  use 

•  Report  generation  was  not  available 

•  Data  was  not  available  or  unreliable 

•  Netscape  vs.  Explorer 

Below  are  some  of  the  more  insightful  comments  on  the  aforementioned  top  five 
concerns.  These  comments  do  not  necessarily  reflect  the  opinions  of  the  authors  and  are 
recorded  as  depictions  of  the  beliefs  of  the  survey  respondents. 

On  the  inability  to  access  LMDSS,  several  reasons  for  the  difficulties  were  listed 
but  the  next  two  comments  best  characterize  the  major  difficulties  encountered. 
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Just  couldn't  access  [LMDSS].  When  I  did  it  stated  ‘Server  Down’  or 
another  time  it  said  that  LMDSS  was  in  the  middle  of  moving... Referred 
me  to  yet  another  site  and  it  stated  it  was  not  available... 

Civilian  Firm  -  Senior  Systems  Analyst 

FIREWALL!  Could  not  fully  access  LMDSS  when  working  off-base. 
Civilian  Firm  -  Analyst 


On  the  ease  of  use,  the  comments  varied  from  specific  to  broad  in  scope.  The 
three  examples  below  provide  a  sense  of  the  overall  picture. 

Too  difficult  to  gain  and  maintain  access  for  occasional  users.  If  you  don't 
use  it  frequently,  you  end  up  not  being  able  to  use  it  at  all. 

USMC  -  LTCOL  ASL-33 

Navigation  to  the  screens  I  was  interested  in  was  NOT  intuitively  obvious. 

By  obvious,  I  mean  that  a  relatively  inexperienced  user  could  not  locate 
the  correct  link  (page)  with  the  first  quick  scan  edit.  I  was  looking  for 
SALTS  information. 

Systems  Command-NAVAIR  Logistics  Info  Systems  Engineer 

...The  method  of  data  extraction  and  the  flexibility  of  output  data  format 
are  no  match  for  the  current  NALDA7ECA  output. 

Civilian  Firm  -  Engineer 


On  report  generation,  several  systems  command  personnel  complained  that 
advertised  report  functions  were  not  available.  One  gentleman  from  a  helicopter  wing 
not  only  had  difficulty  using  the  canned  reports,  but  had  greater  difficulty  trying  to  use 
the  IQ  tool.  He  wrote: 

Although  I  attended  the  IQ  class  here  at  NAS  North  Island  from  a  billet 
assigned  by  AIRPAC,  I  was  never  able  to  get  the  IQ  program  to  work. 
Contacted  CNAP  personnel  and  they  were  having  the  same  problems. 

The  support  at  PAX  River  provided  by  ( name  withheld)  was  useless  at  the 
time  also.  So  I  just  gave  up  until  it  is  fixed.  I  do  need  it  and  reverted  back 
to  doing  reports  from  the  old  NALDA  and  TDSA  databases. .. . 

Wing  -  Configuration  Manager 
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Other  personnel  reported  they  did  not  use  LMDSS  in  the  past  due  to  the 
unavailability  of  data  for  certain  aircraft  type  or  components  or  that  data  they  did  receive 
was  unusable  due  to  unreliability  or  incompatibility  with  older  systems. 

Lastly,  on  Netscape  versus  Explorer,  although  question  7  results  indicated  that 
there  is  little  anticipated  difficulty  foreseen  in  getting  and  installing  the  required  browser, 
there  are  still  some  questions  about  the  decision  itself.  Some  respondents  questioned  the 
reasoning  behind  the  move  to  use  Netscape.  One  squadron  maintenance  Chief  with 
extensive  computer  experience  wrote  this  statement: 

I'd  note  here  that  the  above  requirement  for  Netscape  is  poor  headwork. 

Put  aside  that  it  is  not  in  keeping  with  the  IT-21  "Standard"  that  has  been 
endorsed  by  CINCLANT  and  CINCPAC... it  becomes  an  administrative 
and  logistical  burden  to  marry  oneself  to  a  particular  browser.”  Squadron 
Data  Analyst 

4.  Usage  of  Application 

This  section  focuses  on  how  the  respondents  have  used  LMDSS  in  the  past  and 
how  they  intend  to  use  the  application  in  the  future.  The  survey  questions  that  will  be 
analyzed  in  this  section  include  questions  9-14,  16,  18-19,  and  35-37.  Questions  that 
affect  the  IQ  tool  will  also  be  covered  here. 

The  first  two  questions  in  this  section  were  open-ended.  This  was  an  intentional 
tactic  on  the  part  of  the  authors  to  illicit  unbiased  responses  reflecting  what  the  users 
actually  want  from  the  LMDSS  program.  Although  a  list  of  choices  would  have  been 
easier  to  tabulate,  it  is  limiting  in  nature.  Question  9  was,  “What  kind  of  information  do 
you  expect  to  retrieve  from  LMDSS  when  it  becomes  available?”  Table  1 1  summarizes 
the  respondents’  entries  on  this  question. 

Comments  received  from  this  question  were  more  difficult  to  analyze  and 
categorize  than  previous  questions  due  to  the  extreme  range  and  variety  of  submitted 
answers.  As  before,  many  responded  in  vague  generalities  while  other  responses 
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precisely  explained  what  types  of  data  they  desired  to  receive  from  LMDSS.  Seventy- 
nine  users  entered  comments  for  this  question.  Many  of  the  comments  contained  more 


Category 

Number  of  Responses 

Percentage  of  152 
Respondents 

Entered  comment 

79 

52% 

Don’t  Know 

43 

28  % 

Not  Applicable 

26 

17% 

Do  not  intend  to  use  LMDSS 

2 

1% 

Left  completely  blank 

2 

1% 

Table  11  Summary  of  Respondents  ’  Expectations  About  LMDSS  Information 


Comment  Categories 

Number  of 

Percentage  of 

Responses 

152  Responses 

Component  and  engine  failure  histories 

17 

11% 

Reliability  and  maintainability  data 

11 

7% 

Detailed  cost  analysis  data 

11 

7% 

All  3M  data  for  all  naval  aircraft 

9 

6% 

ID  of  major  reliability  degraders 

7 

5% 

Top  readiness  degraders 

7 

5% 

Trend  analysis  data 

6 

4% 

Historical  maintenance  man-hour  data 

5 

3% 

Repairable  histories 

5 

3% 

Flight  hour  summaries 

4 

3% 

Technical  directive  status  information 

4 

3% 

All  data  currently  available  in  NALDA I 

4 

3% 

Cross  reference  tables  for  WUC,  P/N,  NUN,  UIC 

3 

2% 

Historical  SRC  data 

2 

i% 

Support  equipment  historical  data 

2 

1% 

Complete  engine  histories 

1 

1% 

Drill  down  to  JCN  level 

1 

1% 

Cannibalization  historical  data 

1 

1% 

Table  12  Summary  of  Respondents’  Comments  About  Information  Available  from 

LMDSS 


than  one  desired  type  of  information.  Table  12  provides  a  frequency  breakdown  of  the 
user  requested  information  type  categories. 
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When  available,  the  command  organization  and  job  title  were  included  from  the 
respondent  database  to  provide  insight  into  why  these  specific  requests  were  made. 
These  comments  are  not  all  inclusive  but  were  representative  of  the  sample.  Some 
specific  comments  received  from  the  responses  to  this  question  are  listed  below: 

Cannibalization  data,  supply  data,  NMCS/PMCS  data. 

Wing  -  Data  Analyst 

Would  like  to  track  failure  and  removal  data  for  existing  systems  and  new 
system  applications'  during  initial  installation  and  subsequently  as  a 
fielded  system. 

Systems  Command  -  PMA  Data  Analyst 

Failure  data  of  a  particular  part  number.  Hopefully  this  information  will 
help  me  decide  if  we  want  to  open  up  an  engineering  investigation  when  a 
failure  is  reported  by  the  Fleet. 

Depot  -  Data  Analyst 

Primary  concern  is  failure  data  as  it  relates  to  Depot  repair. 

Systems  Command  -  NAVAIR  Data  Analyst 


Question  10  asked,  “What  information  do  you  require  but  are  unable  to  access  via 
LMDSS?”  Table  13  summarizes  these  results. 


Category 

Number  of  Respondents 

Percentage  of 
Respondents 

Don’t  Know 

71 

47% 

Not  Applicable 

44 

29  % 

Entered  comment 

34 

22% 

Left  completely  blank 

3 

2% 

Table  13  Summary  of  Respondent's  Requirements  from  LMDSS 


Fewer  people  answered  this  question  than  the  previous  one.  Referring  back  to 
Figure  3,  the  Frequency  of  Usage  percent  histogram,  only  thirty-seven  out  of  152 
respondents  reported  using  LMDSS  on  at  least  a  monthly  basis.  It  is  not  unexpected 
therefore  to  find  that  a  large  percentage  of  users  do  not  know  what  LMDSS  can  and 
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cannot  provide.  The  length  of  time  LMDSS  has  been  inoperable  is  also  a  contributing 
factor  toward  the  low  response  rate  for  this  question.  Numerous  statements  reflected  the 
fact  that  LMDSS  was  not  fully  functional  nor  were  its  exact  capabilities  well  known  by 
the  users.  Some  items  requested  by  the  users  responding  to  the  web  survey  are  already 
incorporated  into  LMDSS  such  as: 

•  Historical  data  by  part  number  for  all  Navy  Aircraft 

•  Current  Aircraft  Inventory  Records  for  specific  type  aircraft 

•  OP-20  Data,  AH-1 W  Maintenance  Data,  UH-1N  Maintenance  Data 


Others  requested  data  that  is  currently  contained  in  other  portions  of  the  NALDA 
system.  For  example:  “Info  available  on  NALDA  I ECA  report  733”  was  one  comment. 
Still  others  were  unclear  on  whether  LMDSS  would  have  the  correct  capability  but  made 
suggestions  on  what  information  they  believed  was  needed.  Due  to  the  variety  of  the 
requests,  categorization  by  subject  was  not  possible.  Below  are  some  of  the  more 
articulate  comments  in  random  order: 

We  need  failure  information  at  the  serial  number  level  with  the  JCN 
linked  to  depot  data  showing  the  actual  repair  done.  Also  need  link  to 
aircraft  configuration  information  showing  the  number  of  cycles  (hours, 
landings,  etc)  that  the  component  lasted  between  last  repair  and  failure. 

Depot  -  R&M  engineer 

Work  Unit  Code  (WUC);  NUN  Cross  reference  top  degraders  by  cost 
reliability  and  supportability  mean  time  between  failures  (MTBF)  for 
WSC  (Actual  and  Predicted)  cost  of  maintaining  WUC  service  record  card 
information.  USMC  -  Depot  Logistics  Management  Specialist 

I  sure  could  use  real  time  AIMD  engine  inductions  and  current  status. 

Also  real  time  bare  firewalls.  Although  AEMS  gives  me  this  info,  I  would 
like  one  central  database.  Maybe  LMDSS  already  does  this,  but  only  a 
few  engine  areas  were  up  and  running. 

Type  Command  -  COMNAVAIRRESFOR  PP  Logistics  Manager 
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History  on  components,  aircraft,  engines.  Data  to  use  as  analytical 
processes  to  determine  extent  of  work  within  the  depot.  Review  RCM 
data  throughout  the  responsible  commands  in  the  DMD  reporting  chain. 
Depot 

Haven’t  been  able  to  sort  or  select  by  unit  identification  code  (UIC)  or 
command.  Secondary  data:  Type-Model-Series  (TMS)  of  aircraft,  work 
center,  WUC,  labor  hours,  beyond  capability  of  maintenance  (BCM) 
hours,  etc... 

Fleet  Command  -  CINCPACFLT  Manpower  analyst 

I  would  like  to  have  information  on  the  current  state  of  the  Naval  Aviation 
Fleet  in  terms  of:  corrosion,  fatigue,  cracking,  engine  failures,  accidents, 
corrosion  prevention  programs... 

Systems  Command  -  NAWC  Electrical  Engineer 

Cost  per  flight  hour,  mean  time  to  replace  with  valid  metrics  (PMB  issue), 
mean  time  between  failures  with  valid  metrics  (PMB  issue) 

Type  Command  -  COMNAVAJRPAC 


Question  1 1  asked,  “How  difficult  was  it  to  build  matrix  tables  within  LMDSS?” 
The  question  was  done  in  a  five-point  interval  Likert  type  scale  with  a  score  of  one 
identified  as  “impossible”  and  a  score  of  five  being  classified  as  “easy”.  Results  are 
shown  in  Figure  8.  All  Likert  formatted  questions  are  displayed  in  three  dimensional 
graphs. 

As  shown  in  Figure  8,  the  majority  of  the  respondents  answered  Not  Applicable. 
This  could  be  attributed  to  the  fact  that  the  LMDSS  program  had  not  been  on  line  for  a 
while  or  that  the  users  had  not  used  the  tool  to  build  matrix  tables.  This  could  also  be  an 
indicator  that  more  training  needs  to  be  conducted  on  this  aspect  of  LMDSS’  capabilities. 
The  results  of  those  who  did  respond  are  tabulated  in  Table  14. 
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Matrix  Build  Difficulty  Percent  Histogram 
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Oil  How  tfifficuK  was  it  to  build  matrix  tables  within  LMDSS? 


Figure  8  Matrix  Build  Difficulty  Level  Percent  Histogram 


Category 

Number  of  Responses 

Percentage  of  41 
Respondents  who 
Answered 

1  Impossible 

5 

12% 

2 

13 

32% 

3 

17 

42  % 

4 

4 

10% 

5  Easy 

2 

5% 

Total  who  answered 

41 

Table  14  Matrix  Build  Difficulty  Likert  Results 


Most  people  responded  to  this  question  by  answering  in  the  middle  of  the  five 
point  Likert  scale  but  more  tended  to  lean  towards  difficult  than  easy.  Fewer  than  five 
percent  of  those  who  answered  selected  “easy”. 

Question  12  asked  the  respondent  to  check  all  answers  that  applied  to  the 
following  question,  “Which  LMDSS  modules  did  you  access  the  most?”  Results  are 
tabulated  in  Table  15. 
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Answer 

Number  of 
Respondents 

Percentage  of 
Respondents 

Trend  Analysis 

51 

34% 

Not  applicable 

48 

32% 

Cost  Analysis 

37 

24% 

Management  Analysis 

36 

24% 

Supply  Analysis 

34 

22% 

Reference  Information 

29 

19  %  i 

Detailed  Analysis 

29 

19% 

Engine  Analysis 

28 

18% 

Candidate  Identification 

23 

15% 

Don’t  know 

17 

11% 

Application  Management  Tools 

7% 

Change  Requests 

6 

4% 

Other 

6 

4% 

Feature  Synopsis 

2 

1% 

Total  who  used  at  least  one  area 

87 

57% 

Total  who  Don’t  know  or  NA 

65 

43% 

Table  15  Type  of  Access  Difficulties  Table  Summary 
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Q13  Have  you  ever  used  the  IQ  tool  to  buid  any  custom  queries? 


Figure  9  IQ  Tool  Usage  Percent  Histogram 
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The  next  three  questions  to  be  analyzed,  questions  13, 14,  and  16,  all  focus  on  the 
IQ  tool.  Question  13  was  a  straightforward  question  that  asked,  “Have  you  ever  used  the 
IQ  tool  to  build  any  custom  queries?”  Results  are  shown  in  Figure  9.  As  displayed  in  the 
percent  histogram,  the  majority  of  users  had  not  used  the  IQ  tool.  Only  eleven  people 
surveyed  (7  percent)  reported  using  the  IQ  tool. 

Question  14  asked  “Do  you  think  the  IQ  tool  was  user  friendly?”  This  question 
had  a  total  of  sixteen  respondents  (11  percent)  answer  with  a  yes  or  no  response  and  the 


Q14  Do  you  think  the  >0  tool  ms  user  frien%? 


Figure  10  User  Perceptions  of  IQ  Tool’s  Ease  of  Use  Percent  Histogram 

results  are  shown  in  Figure  10.  Only  two  percent  of  the  IQ  users  felt  the  tool  was 
friendly. 

Question  16  differs  from  the  two  previous  questions  in  that  it  asks  about  the  users' 
beliefs  on  the  importance  of  including  an  IQ  tool  within  the  LMDSS  application. 
Question  16  asked,  “How  important  do  you  consider  the  addition  of  a  structured  ad  hoc 
query  tool  to  LMDSS?”  Results  are  included  in  Table  16.  Figure  1 1  displays  the  percent 
histogram  for  question  16. 
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Category 

Coding 

Number  of 
Responses 

Percentage  of  152 
Respondents 

Extremely  Important 

1 

35 

23% 

Very  Important 

2 

45 

30% 

Somewhat  Important 

3 

20 

13% 

Not  Very  Important 

4 

1 

1% 

Not  At  All  Important 

5 

0 

0.0  % 

Don’t  Know,  NA 

NA 

51  1 

34% 

Table  16  Importance  of  IQ  tool  to  LMDSS  Responses  and  Recoding  Criteria 


IQ  Tool  Percent  Histogram 


Q16  How  important  do  you  consider  the  adtfition  of  a  structured, 
ad  hoc  query  tool  to  LMDSS? 


Figure  11  Importance  of  IQ  Tool  Percent  Histogram 

The  majority  (53  percent)  of  respondents  declared  the  addition  of  an  IQ  tool 
would  be  “extremely”  or  “very  important”.  This  is  an  interesting  finding  in  light  of  the 
fact  that  only  two  percent  of  the  respondents  said  that  they  found  the  existing  ad  hoc 
query  tool  easy  to  use  in  question  14  (Figure  10)  and  less  than  ten  percent  of  the 
respondents  stated  they  had  accessed  the  existing  tool  in  question  13  (Figure  9). 

Question  18  asked,  “Please  list  a  few  of  the  key  decisions  you  would  like  LMDSS 
to  support  when  it  comes  back  online.”  This  open-ended  question  was  designed  to  focus 
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on  the  type  of  major  decisions  LMDSS  needs  to  support.  Most  users  described  the  type 
of  information  they  required  instead  of  describing  the  key  decisions  as  requested.  Table 
17  summarizes  the  results  of  this  query. 


Category 

Number  of  Responses 

Percentage  of  152 
Respondents 

Don’t  Know 

63 

41% 

Not  Applicable 

51 

34% 

Entered  comment 

39 

26  % 

Left  completely  blank 

3 

2% 

Table  1 7  Summary  of  Respondents’  Recommendations  on  LMDSS  Key  Decisions 


Due  to  the  large  variety  of  answers,  categorization  of  this  data  was  not  practical. 
Four  insightful  comments  listed  below  are  representative  of  the  collected  results. 

Life  cycle  cost  decisions,  reliability  centered  maintenance  decisions, 
integrated  maintenance  concept  decisions,  affordable  readiness  decisions, 
defense  systems  affordability  decisions,  supportability  issues,  acquisition 
issues,  etc... 

Depot  -  Mechanical  Engineer/Data  Analyst 

Create  some  simple  tool  to  show,  in  plain  language,  the  present  support 
status  of  a  given  item.  One  page  with  cost,  AYDLR,  items  on  hand,  items 
due  in,  quarterly  demand,  number  of  maintenance  actions  at 
Organizational  and/or  Intermediate  level  for  the  most  recent  twelve 
months,  and  number  of  A-799s  within  those  maintenance  actions  should 
give  the  typical  user  (and  his/her  novice  level  management  audience)  the 
sufficient  data  to  make  educated  decisions. 

Depot  -  Electronics  Engineer 

Part  number  usage  and  part  number  cost  of  ownership  (how  much  does  it 
cost  to  maintain  a  part  number  over  a  specified  time). 

Civilian  Firm 

Is  end  item  experiencing  infant  mortality  or  wear  out  failure  modes?  How 
many  cycles  did  the  unit  operate  before  failure?  What  was  the  specific 
failure  mode?  Is  it  worthwhile  to  invest  in  reliability  improvements  to  a 
given  end  item  (need  cost  and  configuration  data) 

Depot  -  R  &  M  Engineer 
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Question  19  asked,  “LMDSS  will  collect  much  of  its  raw  data  from  NALCOMIS. 
If  there  is  any  data  you  will  require  which  is  not  captured  by  NALCOMIS,  please 
describe  it.”  Table  18  displays  the  summarized  results. 


Category 

Number  of  Respondents 

Percentage  of 
Respondents 

Don’t  Know 

65 

43% 

Not  Applicable 

56 

37  % 

Left  completely  blank 

3 

2% 

Entered  comment 

28 

18% 

Table  18  Summary  of  Respondents’  Items  not  covered  in  NALCOMIS 


Many  comments  in  this  section  reflected  a  lack  of  knowledge  about  the  data 
available  for  retrieval  within  LMDSS.  A  few  comments  referred  to  items  not  currently 
available  within  the  LMDSS  application.  The  other  comments  included  such  topics  as: 

•  Increased  supply  statistics 

•  Support  equipment  data  Technical  Directives  Library  Configuration 
Control  information 

•  Government  On-line  Data  System  (GOLD) 

In  general,  most  respondents  appeared  comfortable  with  the  data  provided  by  the 
Naval  Aviation  Logistics  Command  Management  Information  System  (NALCOMIS), 
provided  the  databases  and  the  matrices  were  properly  formatted  and  accurate. 

Question  35  asked,  “What  might  some  of  your  reasons  be  for  using  LMDSS 
when  it  returns  to  the  web?  (Check  all  that  apply)”.  Results  are  displayed  in  Table  19. 

Question  36  was  an  open-ended  question  which  requested  users  to  clarify  their 
definitions  if  they  checked  the  “other”  category  in  question  35.  Of  the  fifteen  people 
who  selected  “other”  in  question  35,  only  ten  entered  comments.  Three  of  the  responses 
mentioned  manpower  calculations.  Other  selections  are  listed  below: 
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Answer 

Number  of 
Responses 

Percentage  of  152 
Respondents 

Conduct  Trend  Analysis 

115 

76% 

Reliability  information 

104 

68% 

Identify  system  degraders 

100 

66% 

Identify  high  cost  drivers  to  conduct  cost  analysis 

91 

60% 

Find  data  to  support  logistics  acquisition 
decisions 

89 

59% 

Search  for  data  to  complete  periodic  reports  and 
forms 

77 

51% 

Track  the  result  of  implemented  improvements 

76 

50% 

Drill  down  to  the  MAF  level  to  determine  cause 
of  system  degraders 

73 

48  % 

Reduce  life  cycle  support  costs  of  aviation 
systems 

72 

47% 

View  cost  data  and  flight  hour  history  of  aircraft 
engines 

68 

45% 

Measure  the  impact  of  decisions  or  policies  on 
platform  readiness 

61 

40% 

61 

40% 

Compare  the  performance  of  my  command  with 
similar  commands 

34 

22% 

Other  (Please  describe  in  question  36  only) 

15 

10% 

Completely  Blank  Entries 

6 

4% 

Users  who  completed  at  least  one  block 

146 

96% 

Table  19  Users  Reasons  for  Using  LMDSS 


Once  again  the  reasons  for  using  LMDSS  justify  the  need  for  data 
analysis  to  review  raw  data  and  summarize  info  into  a  usable  format  for 
the  APML....  Otherwise  we  will  expend  an  awful  lot  of  time  on  data  that 
may  conflict  with  other  decision-making  systems. 

Systems  command  -  DAPML 

Get  detailed  failure  mode  information  at  the  serial  number  level.  Get 
depot  data  linked  by  Job  Control  Number  (JCN)  to  maintenance  action 
form  (MAF).  Get  operating  cycles  information  at  the  serial  number  level. 
Depot  -  R&  M  Engineer 
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Review  historical  hours  and  workload  /  backlog  to  assist  in  determining 
billet  requirements  for  a  commands  Activity  Manning  Document. 

Fleet  command  -  CINCPACFLT  Management  Analyst 

...  I  have  had  logistics  people  collect  failure  data  of  a  particular  part 
number  to  help  in  my  decision  to  open  up  an  engineering  investigation  on 
the  failed  item. 

Systems  command 

Question  37  asked,  “Referring  to  your  answers  in  questions  #  35  and  #  36,  what 
do  you  anticipate  your  primary  reason  for  using  LMDSS  will  be?”  Results  for  this 
question  are  included  in  Tables  20  and  21.  On  this  question  some  users  repeated  choices 
listed  in  questions  35  or  36  while  others  provided  unique  answers.  Some  of  the  replies 
from  question  37  were: 


Category 

Number  of 
Responses 

Percentage  of  152 
Respondents 

Entered  comment 

79 

52% 

Don’t  Know 

43 

28% 

Not  Applicable 

26 

17% 

Do  not  intend  to  use  LMDSS 

2 

1  % 

Left  completely  blank 

2 

1% 

Table  20  Summary  of  Respondents’  Expectations  About  LMDSS  Information 


Depends  on  training.  Time,  workload,  and  specialization  are  drivers.  If 
training  is  intense  and  concentrated,  i.e.  short  term  and  Frequently  Asked 
Question  (FAQ)/Help  functions  are  improved,  the  primary  reason  for  use 
will  be  to  assist  in  the  acquisition  logistics  determinations  AMD  monitor 
those  decisions  in  sufficient  detail  to  allow  for  change  before  change 
becomes  too  costly. 

Systems  Command  -  APML 

Since  I  have  no  data  analysis  support  I  would  think  this  system  if  accurate 
could  be  of  great  benefit  in  managing  my  engine  program  on  a  day  to  day 
basis. 

Systems  Command  -  Engine  DAPML 
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Long-term  analysis  and  development  of  improved  analysis  techniques  (in 
particular,  studying  predictive  indicators,  similar  in  purpose  to  the 
Emergent  Problems  Matrix). 

Systems  Command  -  Logistics  Management  Specialist 

...developing  composite  systems  for  comparative  analysis,  verifying 
system  performance  and  cost  once  new  acquisition  is  fielded. 

Other  -  ILS  manager 


Answer 

Number  of 
Responses 

Percentage  of  152 
Respondents 

Conduct  Trend  Analysis 

20 

13% 

Reliability  information 

20 

13% 

Identify  system  degraders 

18 

12% 

Identify  high  cost  drivers  to  conduct  cost  analysis 

15 

10% 

Reduce  life  cycle  support  costs  of  aviation 

13 

9% 

systems 

Track  the  result  of  implemented  improvements 

9 

6% 

Measure  the  impact  of  decisions  or  policies  on 

9 

6% 

platform  readiness 

Aircraft  Engine  Data 

7 

5% 

Find  data  to  support  logistics  acquisition 
decisions 

6 

4% 

Technical  Support/TD  compliance 

5 

3% 

Daily  Management 

5 

3% 

Support  the  Fleet 

5 

3% 

Drill  down  to  the  MAF  level  to  determine  cause 

4 

3% 

of  system  degraders 

Search  for  data  to  complete  periodic  reports  and 
forms 

4 

3% 

Measure  the  effect  of  improvement  actions  on 
support  costs 

4 

3% 

View  cost  data  and  flight  hour  history  of  aircraft 
engines 

3 

2% 

Parts  Availability/Tracking 

3 

2% 

Compare  the  performance  of  my  command  with 
similar  commands 

2 

1% 

Table  21  Primary  Reasons  for  Using  LMDSS 


73 


Measure  the  effect  of  improvement  actions  on  support  costs.  This  is  vital 
to  the  Depots  in  reducing  cost  and  continuing  to  produce  a  quality 
product. 

Depot  -  Maintenance  Data  System  Coordinator 


5.  Modeling  and  Simulation 

This  topic  was  covered  in  survey  questions  20  and  21.  The  former  question  was  a 
Likert  scale  formatted  question  that  requested  the  user  to  determine  “How  important  do 


Category 

Coding 

Number  of 
Responses 

Percentage  of 

152  Respondents 

Extremely  Important 

1 

17 

11  % 

Very  Important 

2 

26 

17% 

Somewhat  Important 

3 

30 

20% 

Not  Very  Important 

4 

7 

5% 

Not  at  All  Important 

5 

2 

1  % 

Don’t  Know,  NA 

NA 

70 

46% 

Table  22  Importance  of  Modeling/Simulation  Capabilities 


you  consider  the  development  of  a  modeling/simulation  capability  in  future  versions  of 
LMDSS?”  Table  22  and  Figure  12  demonstrate  the  results  of  this  question. 

From  Table  22,  it  is  obvious  that  nearly  half  of  the  users  answered  “don’t  know” 
or  “not  applicable”.  The  majority  of  those  who  did  voice  an  opinion  were  in  favor  of 
further  development  of  modeling  and  simulation  capabilities.  This  question  yielded  a 
much  higher  response  rate  than  the  similarly  formatted  question  1 1,  Figure  9,  which  was 
based  upon  the  IQ  tool.  This  suggests  a  greater  familiarity  of  modeling/simulation 
techniques  among  users  than  knowledge  about  the  use  of  IQ  tools. 
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Modeling/Simulation  Percent  Histogram 


mm 


Q20  How  important  do  you  consider  the  deeetopment  of  a  modeting/sinujtatian 
capacity  in  future  versions  of  LMDSS? 


Figure  12  Modeling/Simulation  Percent  Histogram 


Category 

Number  of  Responses 

Percentage  of  152 
Responses 

Not  Applicable 

61 

40  % 

No 

59 

39% 

Entered  comment 

17 

33% 

Don’t  Know 

8 

5% 

Did  not  reply 

7 

5  % 

Table  23  Response  Summary  for  Model/Simulation  Examples 


Question  21  asked,  “Can  you  provide  an  example  of  how  a  modeling/simulation 
capability  to  conduct  sensitivity  (what  if)  analysis  will  be  useful?”  Table  23  provides  a 
frequency  breakdown  of  the  users  who  provided  examples  of  modeling/simulations  that 
would  be  useful.  Many  of  the  comments  contained  more  than  one  example.  The  specific 
'  types  of  examples  are  categorized  and  displayed  in  Table  24. 
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Comment  C  ategories 

Number  of 
Responses 

Percentage  of 
152  Responses 

Predict  future  manpower  and  repair  requirements 
and  how  they  are  affected  by  flight  hour  levels 

8 

5% 

Impact  of  A799  removal  levels  on  readiness 

3 

2% 

Estimating  life  cycle  costs  and  impacts  of  aging 

2 

1  % 

Compare  Intermediate  level  costs  to  Depot  level 
costs 

2 

1  % 

Determine  return  on  investment  for  proposed 
readiness  and  maintainability  program  changes 

2 

1  % 

Forecast  spare  costs  and  failure  trends 

2 

1% 

Justify  Reliability  Centered  Maintenance  (RCM) 
and  Engine  Change  Proposal  (ECP)  analysis 

2 

1% 

Compare  historic  costs  to  actual  projected  costs 

1 

1  % 

Table  24  Summary  of  Types  of  Modeling/Simulation  Examples 


Some  of  the  more  pertinent  selections  received  from  the  responses  to  this 
question  are  listed  below: 

For  a  low  cost  system  like  Joint  Direct  Attack  Munitions,  compare 
contractor  depot  vs.  government  depot  vs.  disposable,  considering  costs 
for  movement,  parts  availability,  labor,  etc.  ..  For  a  low  cost  container 
(approximately  $600)  determine  the  break  point  for  defining  'reusable', 
considering  costs  of  movement,  repair,  labor,  materials,  disposal,  etc. 

Civilian  firm  -  Logistics  Analyst 

. . .  perform  some  type  of  trend  analysis  or  probabilistic  modeling  of  the 
factors  influencing  engine  repairs.  This  would  help  me  build  my  engine 
repair  requirements  for  the  out-years  of  the  budget. 

Fleet  Command  -  Depot  Requirements  Officer 

...If  annual  flight  hours  increase  (or  decrease),  how  many  more 
maintenance  man-hours  will  be  required?  If  cannibalization  actions  are 
occurring,  how  many  maintenance  man-hours  have  been  expended? 

Type  Command  -  Analyst 

Life  Cycle  Cost  Estimating  -  Sensitivity  by  changing  anticipated  flight 
horn  usage.  Impacts  of  aging  -  Sensitivity  to  cost  increases  of 
maintenance  over  time. 

Systems  Command  -  Operations  Research  Analyst 
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We  have  to  determine  the  trade-off  of  investment  cost  versus  reliability 
improvement  benefits.  A  model  that  would  link  future  projected  flight 
hours  to  a  population  of  end  items  would  be  a  very  beneficial  tool  for  this 
type  of  analysis. 

Depot  -  Readiness  and  Maintainability  Engineer 

6.  Training 

A  total  of  five  questions,  questions  26  through  30,  will  be  covered  in  this  section. 
The  first  one,  question  26,  requested  information  on  “How  would  you  rate  the  quality  of 
the  LMDSS  instruction  you  have  received  from  NAVAIR  personnel?”  Results  are 
compiled  in  Table  25. 


Categories 

Number  of  Responses 

Percentage  of  1 52 
Respondents 

Excellent 

6 

4% 

Good 

12 

8% 

Fair 

5 

3% 

Poor 

5 

3% 

Don’t  remember 

5 

3% 

Did  not  receive  training 

74 

48% 

Not  applicable 

45 

30% 

Table  25  Quality  of  LMDSS  Training 


Of  the  152  respondents,  only  thirty-three  had  definitely  received  training  on  the 
LMDSS  system  and  nearly  fifty  percent  stated  they  had  not  had  any  training  on  the 
program. 

The  next  question  built  on  this  question  and  asked,  “Did  the  training  include 
Structured  Query  Language  training  using  the  IQ  tool?”  Of  the  thirty-three  who  received 
the  training,  nine  answered  “yes”,  fourteen  answered  “no”,  and  the  remainder  replied 
“don’t  remember”.  Neither  question  probed  into  when  the  user  received  training  so  it  is 
not  possible  to  determine  if  the  respondents  who  answered  “no”  took  their  training  before 
or  after  SQL  training  was  added  to  the  NAVAIR  syllabus. 
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Question  28  asked  the  users  “Do  you  feel  you  will  need  LMDSS  refresher  training 
when  LMDSS  comes  back  online?”  The  majority  of  the  respondents  said  “yes”  (68 
percent)  with  only  eleven  people  answering  “no”  (seven  percent).  The  remaining  thirty- 
four  people  (22  percent)  checked  the  “don’t  know”  block. 

The  following  question  was  developed  directly  from  the  focus  group  that  was 
held  at  NAVAIR.  The  focus  group  participants  wanted  to  know  the  various  levels  of 
importance  users  assigned  to  different  LMDSS  training  strategies.  Question  29  was  a 
Likert  question  that  asked,  “How  important  is  ‘hands  on’  LMDSS  training  with  each 
student  assigned  to  a  computer  terminal?”  This  question  did  not  specify  the  location  of 
the  training  or  whether  the  training  was  to  be  a  classroom  environment  or  one-on-one 
training.  Results  are  shown  in  Table  26  and  Figure  13. 


Category 

Coding 

Number  of 
Responses 

Percentage  of 

152  Respondents 

liiRffjMMMiiiiiiiiiKml 

1 

55 

36.2  % 

2 

39 

25.6  % 

Somewhat  Important 

3 

18 

11.8% 

Not  Very  Important 

4 

2 

1.3  % 

5 

1 

0.7  % 

Don’t  Know,  NA 

NA 

37 

24.3  % 

Table  26  Importance  of  Hands  On  LMDSS  Training 


The  numbers  clearly  show  that  most  users  place  a  high  degree  of  importance  on 
hands  on  training. 

The  final  question  in  this  section  was  an  open-ended  question  asking,  “How  can 
LMDSS  training  be  improved?”  A  summary  of  respondents’  results  is  covered  in  Table 
27  and  a  list  of  means  of  improvements  is  listed  in  Table  28. 
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Figure  13  Importance  of  Hands  On  LMDSS  Training  Percent  Histogram 


Category 

Number  of  Responses 

Percentage  of  152 
Responses 

Not  Applicable 

59 

39% 

Don’t  Know 

58 

38  % 

Entered  Comments 

33 

22% 

Did  not  reply 

2 

1  % 

Table  27  Response  Summary  for  LMDSS  Training  Improvements 


All  respondents  who  stated  that  they  had  received  training  entered  suggestions  on 
how  they  would  improve  LMDSS  training.  User  suggestions  are  categorized  in  Table  28. 
Several  people  made  multiple  suggestions  within  a  single  comment  entry. 
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Comment  Categories 

Responses 

Percentage  of 
152  Responses 

More  hands-on  practice  sessions  with  realistic  data 

13 

9% 

Develop  an  online  Web  based  LMDSS  tutorial 

12 

8% 

Increase  training  availability  to  reach  more  users  in 
a  more  timely  manner 

6 

4% 

Extend  training  period  to  better  cover  material 

4 

3% 

Distribute  CD-ROM  tutorials 

4 

3% 

Develop  two  different  training  courses,  one  for 
supervisors  and  one  for  data  analysts 

3 

2% 

Develop  a  more  adequate  “Help”  function 

2 

1% 

More  ad-hoc  query  training 

2 

1% 

Expand  training  to  include  contractors 

2 

1% 

Table  28  List  of  Recommendations  to  Improve  LMDSS  Training 


Some  specific  comments  received  from  the  responses  to  this  question  are  listed 

below: 


....  I  have  never  seen  any  newsletters  as  to  what  is  going  on  with  the 
program  as  we  did  when  there  was  just  plain  NALDA...I  feel  they  have 
our  email  addresses  and  should  put  out  a  newsletter  as  to  what  is  in  the 
works,  what  some  of  the  problems  are,  and  when  we  can  expect  them  to 
be  resolved. 

Wing  -  Aircraft  Configuration  Manager 

LMDSS  must  be  online  when  training  is  provided...  When  I  took  the  class, 
LMDSS  was  not  ready  for  us  and  in  fact  provided  me  with  a  rather 
negative  perspective  of  its  usefulness. 

Depot  -  Data  Analyst/Engineer 

...  I  was  quickly  given  an  introduction  to  LMDSS  during  a  trip  to  Patuxent 
River  once,  but  never  received  training  after  getting  my  user  ID  and 
password.  Had  trouble  getting  in  and  eventually  gave  up  trying. 

System  Command  -  Program  Analyst 

Tailor  it  for  the  level  of  user.  As  a  PMA,  I  don’t  anticipate  using  many  of 
the  intricate  workings  but  I  want  access  to  check  on  things  occasionally 
and  I  want  it  to  be  simple  enough  that  I  don’t  become  frustrated  in  the 
attempt. 

System  Command  -  PMA 
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Additional  training  in  the  development  and  submittal  of  highly 

individualized  queries. 

Civilian  Firm  -Engineering  Support  Analyst 

7.  Graphics 

Questions  24  and  32  were  developed  to  determine  user  concerns  about  graphical 
capabilities  within  the  LMDSS  program.  Both  questions  were  Likert  scale  questions  but 
each  focused  upon  a  different  aspect.  The  first  question  asked  about  the  development  of 
graphical  capabilities  within  the  LMDSS  system;  whereas,  the  second  question  queried 
the  user  about  the  need  to  develop  the  means  to  cleanly  export  retrieved  data  to  already 
existing  software  applications.  It  should  be  noted  that  both  of  these  two  questions 
generated  high  responses.  Question  24  had  an  80.9  percent  response  rate  and  question  32 
received  an  83.6  percent  rate. 

Specifically,  question  24  asked,  “How  important  do  you  consider  the 
development  of  a  graphics  capability  in  future  versions  of  LMDSS?”  The  results  are 
shown  in  Table  29  and  Figure  14.  In  question  24,  the  largest  group  of  users  listed 
internal  graphical  capabilities  as  “somewhat  important”  but  more  users  selected  positive 
responses  than  negative  ones 


Category 

Coding 

Number  of 
Responses 

Percentage  of 
152  Respondents 

Extremely  Important 

1 

24 

16% 

Very  Important 

2 

32 

21% 

Somewhat  Important 

3 

50 

33% 

Not  Very  Important 

4 

15 

10% 

Not  at  All  Important 

5 

3 

2% 

Don’t  Know,  NA 

NA 

29 

19% 

Table  29  Importance  of  Graphical  Capabilities  within  LMDSS 
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024  How  important  do  you  consider  the  development  of  «  graphics  capacity 
in  future  versions  of  LMDSS  ? 


Figure  14  Importance  of  Graphical  Capabilities  within  LMDSS  Percent  Histogram 

The  next  question  asked,  “How  important  is  it  for  future  versions  of  LMDSS  to  be 
able  to  cleanly  export  data  to  applications  with  graphics  capabilities  such  as  Excel?” 


Category 

Coding 

Number  of 
Responses 

Percentage  of 

152  Respondents 

Extremely  Important 

1 

61 

40% 

Very  Important 

2 

45 

30% 

Somewhat  Important 

3 

20 

13% 

Not  Very  Important 

4 

1 

1% 

Not  at  All  Important 

5 

0 

0.0  % 

Don’t  Know,  NA 

NA 

25 

16% 

Table  30  Importance  of  Graphical  Export  Capabilities 
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This  Likert  question  received  the  most  positive  response  in  the  entire  survey  as  shown  by 
the  data  displayed  in  Table  30  and  Figure  15.  Nearly  seventy  percent  of  all  respondents 
stated  it  was  extremely  important  or  very  important  to  have  good  export  capabilities  to 
graphical  applications. 

8.  Users’  Perceptions  about  the  LMDSS  application 

This  section  contains  the  questions  that  were  perhaps  the  most  difficult  to 
measure  and  analyze  because  they  did  not  attempt  to  collect  facts  and  figures.  The 
questions  asked  here  were  used  to  ascertain  the  beliefs  and  perceptions  of  the  users 
toward  the  LMDSS  system.  Since  this  type  of  information  is  normally  difficult  to  collect 
using  a  survey  tool,  key  questions  were  asked  in  several  different  formats:  yes/no 
questions,  Likert  type  scale  questions  and  open-ended  questions.  Many  questions  were 
similar  to  try  to  obtain  subtle  nuances.  The  questions  covered  in  this  section  include 
questions  15,  17, 22-23, 25, 33-34, 38-39. 
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The  first  perception  question  asked  was  a  simple  yes/no  question  inserted  at  the 
end  of  the  Usage  section.  The  straightforward  question  15  asked,  “Do  you  think  LMDSS 
was  user  friendly?”  Nearly  half  of  the  replies  were  in  the  “don’t  know”  category  (49 
percent).  Those  who  gave  a  yes  or  no  answer  were  evenly  split  with  forty  saying  “yes” 
(26  percent)  and  thirty-six  saying  “no”  (24  percent).  Again  the  large  group  of  people 
answering  “don’t  know”  may  have  been  due  to  the  extended  period  of  time  when  the 
application  was  not  in  working  order.  Overall,  these  results  suggest  there  is  still  room 
for  improvement  in  this  area. 

Question  17  asked,  “If  you  know  of  other  NALDA  applications  which  contain 
query  formats  you  would  like  to  see  implemented  in  LMDSS  please  list  them  below?” 
Table  31  summarizes  the  results  of  this  query. 


Not  Applicable 

130 

86% 

Don’t  Know  or 

Left  completely  blank 

Entered  comment 

22 

14% 

Table  31  Summary  of  Respondents’  Recommendations  about  Other  NALDA  Formats 


The  only  general  theme  appearing  among  most  of  these  responses  was  simplicity. 
Users  did  not  appear  to  hold  strong  opinions  about  queiy  format  as  long  as  it  was  easy  to 
leam  and  apply.  The  Aircraft  Inventory  and  Readiness  Reporting  System  (AIRRS)  and 
the  Aircraft  Engine  Management  System  (AEMS)  were  the  only  two  NALDA 
applications  that  received  at  least  three  or  more  recommendations.  One  worthwhile 
comment  came  from  an  ILS  manager  from  SPA  WAR  who  suggested,  “No  NALDA 
applications  come  to  mind,  but  you  could  apply  the  NSLC  (shipboard)  equivalent  called 
"3M  OARS".  That  application  is  very  user  friendly  with  ad-hoc  reporting,  conversion  to 
commercial  database  software  and  web  access.” 

Questions  22  and  23  were  Likert  type  questions  developed  to  assess  how  the  users 
perceived  the  underlying  structure  of  the  software  application  program.  The  first 
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question  targeted  how  much  importance  the  user  attached  to  knowing  the  structure  of  the 
report  generation  formulas  and  the  second  queried  the  usefulness  of  having  an  imbedded 
data  dictionary. 

The  results  of  question  22,  “How  important  is  it  for  you  to  know  the  details 
concerning  how  LMDSS  reports  are  derived  before  you  feel  you  can  trust  the  information 
they  provide?”  are  displayed  in  Table  32  and  Figure  16. 


Category 

Coding 

Number  of 
Responses 

Percentage  of 
152  Respondents 

Extremely  Important 

1 

48 

32% 

2 

34 

22% 

Somewhat  Important 

3 

29 

19% 

Not  Very  Important 

4 

10 

7% 

iSBBSB 

5 

0 

0% 

Don’t  Know,  NA 

NA 

31 

20% 

Table  32  Importance  of  LMDSS  Report  Derivations 


022  How  important  is  it  for  you  to  know  the  details  concerrang  how  LWDSS  reports  are 
derived  before  you  fee!  you  can  trust  the  Information  they  provide? 


Figure  16  Importance  of  LMDSS  Report  Derivations  Percent  Histogram 
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Nearly  eighty  percent  of  all  respondents  entered  an  opinion  for  question  22  and 
over  fifty  percent  felt  that  it  was  “extremely”  or  “very  important”  to  understand  how  data 
are  derived  by  LMDSS  in  order  to  trust  the  data.  This  question  does  not  address  the 
reason  why  users  believe  this  but  it  would  be  interesting  to  investigate  the  relationship 
between  this  question  and  the  users  reported  inability  to  access  the  program  reliably. 


Category 

Coding 

Number  of 
Responses 

Percentage  of 

152  Respondents 

Extremely  Important 

1 

57 

38% 

Very  Important 

2 

49 

32% 

Somewhat  Important 

3 

20 

13% 

4 

5 

3% 

Not  at  All  Important 

5 

0 

0% 

Don’t  Know,  NA 

NA 

21 

14% 

Table  33  Importance  of  LMDSS  Data  Dictionary 


LMDSS  Data  Dictionaiy  Percent  Histogram 


Q23  How  important  is  a  detailed  description  or  definition  (data  dictionary) 

or  all  LMDSS  data? 


Figure  1 7  Importance  of  LMDSS  Data  Dictionary  Percent  Histogram 
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Question  23  queried,  “How  important  is  a  detailed  description  or  definition  (data 
dictionary)  of  all  LMDSS  data?”  The  results  are  displayed  in  Table  33  and  Figure  17. 

Question  25  was  another  straightforward  question  asking,  “Have  you  found  the 
usage  of  definitions  and  metrics  by  LMDSS  to  be  in  compliance  with  established  3M 
definitions?”  Since  usage  was  measured  to  be  low  in  Figure  4,  a  large  number  of 
meaningful  responses  was  not  expected.  The  results  supported  this  expectation  and  are 
displayed  in  Table  34. 


Category 

Number  of  Responses 

Percentage  of  152 
Respondents 

Yes 

32 

21  % 

No 

5 

3% 

Don’t  Know 

113 

74% 

Table  34  Compliance  with  3M  Definitions  Summary  Results 


Question  31  requested  users  to  “Please  describe  any  examples  of  your  not  being 
satisfied  with  the  quality  of  the  data  you  have  obtained  through  LMDSS."  The  summary 
results  are  provided  in  Table  35.  Many  of  the  comments  contained  more  than  one 
example.  Table  36  provides  a  frequency  breakdown  of  types  of  user  answers. 


Category 

Number  of  Responses 

Percentage  of  152 
Responses 

Not  Applicable 

92 

60  % 

None 

24 

16% 

Don't  Know 

9 

6% 

Did  not  reply 

3 

2% 

Entered  comment 

24 

16% 

Table  35  Summary  of  Examples  of  Dissatisfaction  with  Quality  of  LMDSS  Data 
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Comment  Categories 

Number  of 
Responses 

Percentage  of 
152  Responses 

LMDSS  data  not  current  or  accurate 

16 

11% 

Difficult  to  filter  and  obtain  desired  data 

6 

4% 

Poor  quality  of  3M  data  from  organizational  level 

5 

3% 

Unable  to  access  desired  modules 

4 

3% 

No  data  for  desired  aircraft  type,  model,  series 

4 

3% 

LMDSS  failure  data  did  not  agree  with  older  systems 

2 

1% 

Cost  per  flight  hour  calculations  erroneous 

2 

1% 

Table  36  Summary  of  Respondents *  Comments 


Some  specific  comments  received  from  the  responses  to  this  question  are  listed 

below. 


Couldn't  get  the  right  cut  of  engine  information.  It  was  simply  too  hard  to 
figure  out  how  to  get  what  I  wanted. 

USMC  -  Maintenance  Officer 

When  data  is  normalized  by  flight  hours  it  often  is  presented  as  all  zeros. . . 
Systems  Command  -  Logistics  Management  Specialist 

...I  was  never  able  to  get  into  many  engine  analysis  areas.  Half  the  area 
was  always  under  repair. 

Type  Command  -  Logistics  Analyst 

Was  told  (LMDSS)  data  was  inaccurate,  so  did  not  use  to  a  great 
degree. . .  I  did  compare  F404  engine/module  removals  with  AEMS  and  the 
numbers  were  off  (engine  transaction  report  data  base  did  not  agree  with 
your  NALCOMIS  3M  data  base 

Civilian  Firm  -  Engine  analyst 

Depot  maintenance  data  at  the  present  time  is  inaccurate  because  of 
limited  data  being  received  from  level  1  and  2  in  the  Depot. . . 

Depot  -  Maintenance  Coordinator 

On  the  few  occasions  I've  used  LMDSS,  the  databases  and  info  I  needed 
were  not  loaded  or  current. 

Civilian  Firm  -  Systems  Analyst 
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I  have  no  formal  training  of  this  system  and  found  it  impossible  to  use.  I 
have  visited  six  AIMDs  and  only  two  AZs  were  competent  in  drawing 
down  data. . .  CNAP  maintains  their  own  data  base  on  site. 

Fleet  Command  -  Management  Analyst  Manager 

To  be  honest,  I  got  frustrated  using  the  LMDSS  system.  It  did  not  provide 
the  information  that  I  would  need  at  the  time  so  I  usually  built  my  own 
NALDA  S2K  queries  and  pull  down  my  data  in  my  that  way. 

Depot  -  Analyst 

Questions  33  and  34  asked,  “Has  anyone  from  NAVAIR  or  the  LMDSS 
development  team  ever  contacted  you  requesting  your  input/feedback  concerning 
LMDSS?”  and  “Have  you  ever  attended  a  NAVAIR  sponsored  LMDSS  users’  group 
meeting?”  respectively.  Twenty  eight  users  (18  percent)  responded  that  they  had  been 
contacted  for  input  concerning  LMDSS  but  only  five  people  (three  percent)  reported 
attending  a  LMDSS  users’  group  meeting.  Of  those  who  attended,  two  were  from  depot, 
one  was  from  a  civilian  firm,  one  was  from  COMNAVAIRRESFOR  and  one  was  from  a 
systems  command. 

The  open-ended  question  38  asked,  “In  what  ways  could  NAVAIR  make  analysts 
more  aware  of  LMDSS?”  Table  37  summarizes  the  responses  to  this  question.  Fifty 
users  entered  comments  for  this  question.  Many  of  the  comments  contained  more  than 
one  recommendation.  Table  38  provides  a  frequency  breakdown  of  how  many  users 
suggested  each  strategy  NAVAIR  could  employ  to  make  analysts  aware  of  LMDSS. 


Category 

Responses 

Percentage  of  152 
Responses 

Don’t  Know 

56 

37% 

Entered  comment 

50 

33% 

Not  Applicable 

42 

28  % 

Did  not  reply 

4 

3% 

Table  37  Summary  of  Suggestions  on  How  to  Increase  Awareness  of  LMDSS 
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Comment  Categories 

Responses 

Percentage  of 
152  Responses 

Training  at  field  activities  and  ‘A’  schools 

20 

13% 

Email 

7 

5% 

Web  page 

6 

4% 

Word  of  mouth 

6 

4% 

Naval  message  traffic 

5 

3% 

Chain  of  command 

3 

2% 

3 

2% 

NAVAIR  newsletter 

3 

2% 

Periodic  LMDSS  user  conferences 

1 

1% 

Table  38  Summary  of  Types  of  Suggestions  on  Increasing  LMDSS  Awareness 


Some  specific  comments  received  from  the  responses  to  this  question  are  listed 


below: 


If  you  build  it  they  will  come! 

NAVICP  systems  analyst 

Email  notices... maybe  like  the  NAVAIR  Team  Newsletters  or  the  BPR 
announcements  that  periodically  show  up  in  our  emails. 

Systems  Command  -  Operations  Research  analyst 

Most  people  have  been  aware  of  LMDSS  or  the  lack  of  LMDSS  for  years. 
If  it’s  up  and  running  awareness  will  take  care  of  itself. 

Systems  Command  -  NAVAIR  Deputy  Department  Head 

Get  it  up  and  running. . .  Ensure  data  correctness.  Let  people  access  it  and 
the  word  will  spread  if  it’s  good. 

Civilian  firm  -  System  Analyst 

Have  a  LMDSS  user  conference  yearly.  The  group  meeting  could  be  run 
similar  to  the  old  NALDA  user  group  meetings.  Discuss  problems, 
concerns,  and  recommendations  for  a  better  system. . . 

Depot  -  Analyst 
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C.  ESTABLISHING  DIFFERENCE  NULL  HYPOTHESES 


In  order  to  answer  the  difference,  or  user  group  comparison,  questions  described 
in  the  previous  chapter,  the  following  null  hypotheses  were  developed: 

•  Hoi  There  is  no  significant  difference  between  the  user  category  groups  (officer, 

enlisted,  GS,  civilian)  with  respect  to  the  importance  they  assign  to  a 
modeling/simulation  capability  in  fiiture  versions  of  LMDSS. 

•  He,:  There  is  no  significant  difference  between  the  organization  type  groups  (Fleet 

command.  System  command.  Type  command,  etc.)  with  respect  to  the 
importance  they  assign  to  a  modeling/simulation  capability  in  future  versions  of 
LMDSS. 

•  Ho!  There  is  no  significant  difference  between  the  user  category  groups  in  how 

important  they  consider  a  graphics  capability  to  be  in  future  LMDSS  updates. 

•  Hq!  There  is  no  significant  difference  between  users  in  the  various  organization  types 

concerning  how  important  they  view  a  graphics  capability  to  be  in  future  LMDSS 
updates. 

•  Hoi  There  is  no  significant  difference  between  user  category  groups  with  respect  to 

the  importance  assigned  to  being  able  to  cleanly  export  LMDSS  data  to 
applications  with  graphics  capabilities  such  as  Excel. 

•  Hq!  There  is  no  significant  difference  between  users  within  the  various  organization 

types  concerning  the  importance  they  assign  to  being  able  to  cleanly  export 
LMDSS  data  to  applications  with  graphics  capabilities  such  as  Excel. 

•  Hq:  There  is  no  significant  difference  between  user  category  groups  concerning  the 

importance  assigned  to  the  addition  of  a  structured,  ad-hoc  query  tool  to  the 
LMDSS  application. 

•  Ho!  There  is  no  significant  difference  between  the  organization  types  concerning  the 

importance  assigned  to  the  addition  of  a  structured,  ad-hoc  query  tool  to  the 
LMDSS  application. 

•  Hq:  There  is  no  significant  difference  between  the  user  category  groups  in  how  often 

LMDSS  is  accessed  via  the  Web. 

•  Hq!  There  is  no  significant  difference  between  the  users  within  the  various 

organization  types  in  how  often  LMDSS  is  accessed  via  the  Web. 
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D.  ANOVA  TESTS 


1.  Meeting  ANOVA  Criteria 

A  one-way  ANOVA  was  used  to  compare  the  means  of  sample/groups  in  order  to 
test  for  significant  differences.  ANOVA  assumes  the  DV  is  interval  scale,  normally 
distributed,  and  the  variances  of  the  groups  are  equal.  If  the  assumptions  are  violated,  an 
equivalent  nonparametric  test  called  the  Kruskal-Wallis  (K-W)  test  can  be  used  to 
compare  the  mean  ranks  of  the  groups.  The  IVs  in  this  study  are  represented  as  ordinal 
data  while  the  DVs  were  measured  on  an  interval  scale.  The  distributions  of  each  of  the 
five  DVs  measured  were  compared  to  a  normal  distribution  curve  to  ensure  they 
approximated  the  distribution  of  a  normal  curve.  The  equality  of  variances  was  tested 
using  the  Levene  test.  If  the  Levene  statistic  was  significant  (sig  <  .05),  the  assumption 
of  equal  variances  was  violated  and  the  K-W  test  was  conducted  to  compare  with  the 
ANOVA  results.  (Morgan  and  Griego,  1998) 

SPSS  software  provides  a  metric  called  the  significance  level,  or  sig,  which 
allows  the  researcher  to  easily  interpret  a  calculated  statistic  in  a  computer  generated 
output  without  looking  up  the  critical  value  in  a  table.  Sig  can  be  considered  to  be  the 
probability  of  a  Type  I  error.  A  Type  I  error  is  the  likelihood  of  rejecting  the  null 
hypothesis  when  it  is  actually  true.  If  the  reported  sig  is  less  than  .05  the  finding  is 
statistically  significant  and  the  null  hypothesis  (no  difference)  is  rejected.  (Morgan  and 
Griego,  1998) 

The  ANOVA  statistic  only  compares  means  between  groups  and  is  used  to 
identify  a  statistically  significant  difference  between  means.  If  the  overall  F  value 
calculated  by  an  ANOVA  test  is  significant  (sig  <  .05)  then  there  is  a  statistically 
significant  difference  between  groups  with  respect  to  the  DV.  Any  further  information 
comparing  the  means  to  each  other  is  done  with  post  hoc  tests.  Post  hoc  tests  are  only 
done  if  the  overall  F  is  found  to  be  significant. 
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2. 


ANOVA  Testing  Results 


ANOVA  testing  of  each  of  the  ten  null  hypothesis  did  not  reveal  any  statistically 
significant  differences  among  various  user  categories  and  organization  type  groups 
concerning  the  measurements  of  the  DVs.  In  the  three  instances  where  the  Levene  Test 
revealed  the  violation  of  the  equal  variances  ANOVA  assumption,  the  K-W 
nonparametric  test  was  conducted  to  see  if  a  different  result  could  be  attained.  In  all 
three  cases  the  K-W  test  results  agreed  with  the  ANOVA  results.  None  of  the  null 
hypotheses  could  be  rejected. 

Specific  pairs  of  organization  type  groups  were  selected  for  ANOVA  testing  to 
determine  if  a  difference  could  be  detected  with  respect  to  measurements  of  the  DVs  and 
none  was  detected.  All  thirteen  organization  types  were  combined  into  three  groups:  (1) 
higher  echelon  military  commands,  (2)  lower  echelon  military  commands,  and  (3)  private 
contractors.  Subsequent  ANOVA  testing  did  not  reveal  any  evidence  supporting  the 
rejection  of  the  null  hypotheses  of  no  differences  between  the  groups.  Table  39  provides 
a  summary  of  the  results  of  the  one-way  ANOVA  testing  conducted  on  the  ten  null 
hypothesizes.  Since  none  of  the  ANOVA  testing  resulted  in  a  significant  difference 
between  group  means,  post  hoc  tests  were  not  conducted. 

E.  PHONE  INTERVIEW  FINDINGS 

Seventeen  phone  interviews  were  conducted  from  a  randomly  selected  sample  of 
25  LMDSS  users.  The  major  themes  common  to  the  majority  of  the  interviews  were: 

•  Many  expressed  a  strong  desire  for  accurate  and  timely  data. 

•  Users  want  to  be  able  to  conduct  effective  ad-hoc  queries  on  their  own. 

•  More  training  is  needed  on  the  use  of  the  ad-hoc  IQ  tool  within  LMDSS. 

•  The  capability  to  drill  down  to  the  Maintenance  Action  Form  (MAF)  level 
is  considered  critical  in  investigating  maintenance  and  logistics  issues. 
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11„ 

(IV  and  DY) 

Calculated 

F  value  for 
ANOVA 

ANOVA  Sig 
Value 

K-\V  Sig  Value 

Can  Il„  be 
Rejected? 

ES9 

.977 

Levene  Test  not 
significant 

NO 

.973 

NO 

.334 

Levene  Test  not 
significant 

NO 

F(12.111)  =  .93 

.412 

Levene  Test  not 
significant 

NO 

User  Cat  and 
Data  Export 

.412 

Levene  Test  not 
significant 

NO 

Org  Type  and 
Data  Export 

mi 

.529 

.496 

NO 

User  Cat  and 
Ad-Hoc  Tool 

BUH 

.338 

Levene  Test  not 
significant 

NO 

Org  Type  and 
Ad-Hoc  Tool 

F(12,88)  =  1.14 

.340 

Levene  Test  not 
significant 

NO 

User  Cat  and 
Freq  of  Use 

WBBBm 

.433 

.507 

NO 

User  Cat  and 
Freq  of  Use 

.316 

.181 

NO 

Table  39  ANOVA  and Kruskal-Wallis  Test  Results 


•  Most  users  interviewed  had  little  experience  or  knowledge  of  models, 
simulations,  and  sensitivity  analysis  and  were  not  interested  in  them  at 
their  level. 

•  Analysts  want  to  understand  how  data  are  derived  to  include  any 
assumptions,  definitions,  and  algorithms. 

•  Most  users  interviewed  expressed  a  lack  of  confidence  in  the  accuracy  and 
timeliness  (two  to  three  months  late)  of  AV-3M  data. 

•  Many  users  expressed  their  frustration  in  knowing  very  little  about  the 
various  data  analysis  software  tools  developed  by  NAYAIR. 

•  Several  users  recommended  teaching  students  at  both  the  Data  Analyst 
and  A  schools  about  LMDSS  and  other  NALDA II  applications. 
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•  The  majority  of  users  recounted  an  unsatisfactory  experience  with 
LMDSS  as  a  result  of  difficulties  with  queries  and  limited  amounts  of 
data. 

•  Those  familiar  with  the  IQ  tool  found  it  extremely  complex,  requiring 
intensive  attention  to  detail. 

•  Most  of  the  communication  between  the  LMDSS  users  interviewed  and 
NAVAIR  occurs  during  sessions  between  the  NAVAIR  3.6.2  help  desk 
personnel. 

•  None  of  the  users  reported  being  contacted  specifically  for  user  feedback 
concerning  LMDSS. 

•  All  users  contacted  had  some  form  of  a  Pentium  computer  on  their  desk 
with  a  way  of  accessing  the  Web. 

•  The  majority  of  logistics  analysts  interviewed  wanted  the  capability  to 
track  the  parts  much  like  FEDEX  and  UPS  tracks  shipments  in  the 
commercial  world.  Currently,  finding  the  status  of  an  ordered  part,  its 
location,  and  who  controls  its  distribution  takes  a  day’s  worth  of  phone 
calls  to  NAVICP.  Several  users  on  WING  and  TYCOM  staffs  considered 
this  an  important  capability. 

Some  statements  recorded  from  those  interviewed  were  as  follows: 

Using  LMDSS  needs  to  be  as  easy  as  following  the  bouncing  ball! 

Email  is  the  best  way  for  NAVAIR  to  stay  in  touch  with  customers  at  all 
levels.  Email  is  a  religion  here. 

I'd  like  one  big  database  in  the  sky  which  lets  me  choose  the  data  elements 
I  need. 

LMDSS  is  not  very  user  friendly.  I  got  lost  trying  to  do  queries.  It  needs 
to  be  more  user  friendly  like  Optimize  NALCOMIS. 

I  accessed  LMDSS  almost  daily  in  the  Fall  of  '98  for  data  queries.  I  was 
not  aware  the  data  within  LMDSS  was  corrupted  and  the  database  had 
been  frozen! 

I've  served  15  years  in  the  Navy  at  the  organizational  level  and  I  never 
heard  of  NALDA  until  I  got  to  the  TYCOM  staff. 
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LMDSS  has  not  progressed  enough  to  allow  me  to  write  my  own  queries 
like  I  did  under  NALDA  I.  LMDSS  doesn't  have  the  depth  of  the  canned 
reports  I  accessed  under  NALDA  I. 

We  developed  our  own  program  called  ’Forecast’  which  uses  NAVFLIRS 
and  comptroller  info  to  conduct  cost  analysis,  cost  comparison,  and 
degrader  identification.  Forecast  can  supply  data  within  10  days  instead 
of  the  three  months  it  takes  LMDSS. 

I  have  been  an  active  proponent  of  the  technology  change  offered  by 
LMDSS  since  its  idea  and  conception  years  ago,  but  there  is  not  any 
usable  information  there  for  me  to  provide  my  customers  so  I  am  not  a 
user. 

F.  INTERVIEWS  WITH  PAST  AND  PRESENT  LMDSS  MANAGERS 

During  the  period  of  this  research,  most  of  the  managers  within  NAVAIR  3.6.2 
were  interviewed  both  in  person  and  over  the  phone.  Some  individuals  no  longer 
assigned  to  NAVAIR  3.6.2  were  also  located  and  interviewed  by  email  and  phone  calls. 
The  major  findings  of  these  interviews  are  summarized  below: 

•  No  evidence  could  be  found  suggesting  the  use  of  automated  software 
tools  for  documentation,  test  cases,  defect  tracking,  and  configuration 
management. 

•  Little  documentation  exists  which  accurately  describes  the  eight  year 
history  of  the  LMDSS  project  in  a  manner  capable  of  supporting 
meaningful  "lessons  learned"  training.  There  is  a  lack  of  historical 
software  measurement  data. 

•  A  formal  risk  management  program  documenting  threats  and  strategies  to 
minimize  them  has  never  been  conducted. 

•  LMDSS  has  undergone  several  major  requirements  changes  as  well  as  at 
least  four  program  manager  changes. 

•  Milestones  and  deadlines  have  been  established  and  reestablished  several 
times  during  the  history  of  the  LMDSS  program. 

•  The  use  of  8A  contractors  with  limited  technical  capabilities  and  a  poor 
understanding  of  LMDSS  requirements  resulted  in  years  of  delays  and 
rework. 
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•  The  LMDSS  project  is  often  funded  out  of  hide  and  has  never  had  its  own 
line  of  funding.  LMDSS  has  suffered  more  than  one  50  percent  budget  cut 
during  its  development. 

•  Estimates  of  the  time  required,  man-hours,  and  size  of  LMDSS  related 
work  was  determined  by  dividing  available  funding  by  the  cost  of 
available,  qualified  programmers. 

•  Changes  in  programming  languages  required  extensive  rework  of  the 
canned  reports. 

•  Most  LMDSS  requirements  are  developed  by  maintenance  and  logistics 
experts  on  the  NAVAIR  staff.  A  robust  strategy  of  involving  LMDSS 
users  from  the  Fleet  to  provide  feedback  and  input  has  not  been 
implemented. 
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VL  DISCUSSION  AND  ANALYSIS 


A.  WHERE  IS  LMDSS  NOW? 

In  December  1998,  LMDSS  was  taken  offline  to  correct  data  error  problems  and 
load  a  minimum  of  eighteen  months  of  AV-3M  data  for  all  types  of  Navy  aircraft.  The 
goal  was  to  get  LMDSS  back  online  within  a  few  months  and  complete  QA  testing  by  the 
scheduled  NALDA  I  termination  deadline  of  March  1999.  This  deadline  was  primarily 
motivated  by  a  DISA  mandate  that  all  systems  be  Y2K  compliant  or  terminated  by  March 
1999.  Continuing  high  data  error  rates  and  problems  loading  the  AV-3M  data  resulted  in 
further  delays  in  completing  the  LMDSS  QA  testing.  Finally,  NAVAIR  3.6.2  directed 
LMDSS  be  brought  back  online  beginning  May  1999  while  QA  testing  continued. 
Although  LMDSS  is  now  back  online  and  considered  operational,  only  four  aircraft 
platforms  (E-2,  H-60,  S-3,  A-6)  are  loaded  as  of  August  1999. 

Currently,  NAVAIR  is  continuing  to  struggle  with  loading  the  data  into  the 
underlying  database  structure.  Data  loads  must  be  completed  before  all  interface 
connections  can  be  properly  reviewed  for  integrity.  Parallel  operations  with  both 
NALDA  I  and  II  servers  are  scheduled  to  continue  during  the  months  of  September  and 
October,  1999.  In  addition,  there  are  three  major  short-term  goals  still  unaccomplished 
by  LMDSS  developers: 

•  Load  at  least  eighteen  months  of  data  for  all  type  equipment  codes. 

•  Successfully  implement  the  automated  update  process  using  AV-3M  data 
from  the  Streamline  Alternative  Logistics  Transmission  System  (SALTS). 

•  Reduce  the  high  data  error  rate  within  the  databases  that  feed  LMDSS. 

Until  these  objectives  are  accomplished,  the  usefulness  of  LMDSS  as  an  effective 
management  information  system  will  be  severely  limited. 
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B.  RESEARCH  QUESTIONS 


This  section  applies  the  findings  of  our  research  in  answering  the  research 
questions  introduced  in  earlier  chapters. 

1.  What  can  future  software  developers  learn  from  studying  the  LMDSS 
project? 

A  well-documented  history  of  the  eight-year  LMDSS  project  is  important  because 
it  will  provide  invaluable  lessons  for  future  NAVAIR  development  teams.  Many  of  the 
classic  mistakes  described  in  the  Literature  Review  and  Appendices  of  this  thesis  were 
identified  during  interviews  with  selected  LMDSS  program  managers.  The  actual 
LMDSS  software  programmers  currently  contracted  by  NAVAIR  were  not  interviewed 
during  this  research.  Additional  interviews  with  these  programmers,  their  predecessors, 
and  a  few  of  the  previous  program  managers  not  contacted  will  be  required  in  order  to 
capture  a  more  complete  LMDSS  development  history. 

The  most  significant  factors  that  contributed  to  delays  in  the  LMDSS 
development  effort  included  the  following: 

•  Absence  of  a  formal  risk  management  program 

•  Major  requirements  changes 

•  Lack  of  funding  support 

•  Proper  contracting  support 

•  New  technology 

•  Lack  of  user  input 

•  Personnel  turnover 

A  more  detailed  summary  of  additional  software  development  mistakes  that 
occurred  during  the  LMDSS  project  is  documented  in  section  F  of  the  Findings  chapter. 
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a)  Formal  Risk  Management 

A  formal  risk  management  program  specifically  addressing  the  LMDSS 
application  was  never  implemented  by  NAVAIR  during  the  history  of  the  LMDSS 
project.  The  literature  review  described  formal  risk  management  as  a  proven, 
fundamental  software  development  practice.  The  early  identification  of  risks,  evaluation 
of  risks,  and  subsequent  implementation  of  strategies  to  minimize  the  impact  upon 
software  development  project  is  critical  in  the  avoidance  of  major  obstacles. 

b)  Major  Requirements  Changes 

Transitioning  LMDSS  from  a  UNIX  to  a  Web  environment  resulted  in 
significant  LMDSS  requirement  changes.  A  formal  risk  management  program  might  have 
fostered  additional  communication  and  planning  which  would  have  allowed  developers 
to  be  better  prepared  for  the  impact  of  these  changes. 

The  majority  of  the  major  requirement  changes  experienced  by  the 
LMDSS  program  were  responses  to  the  exponential  rate  of  improvements  in  networking, 
computers,  and  Internet  technology  within  the  last  few  years.  Additional  changes  to  the 
LMDSS  project  included  the  abandonment  of  the  Ada  and  ActiveX  codes  for  the  newer 
Oracle  and  Java  languages.  Future  development  teams  should  study  how  these  changes 
were  implemented  to  learn  the  importance  of  proactive  planning  and  estimating  using 
formal  risk  management  and  software  metrics. 

c)  Funding  Support 

LMDSS  has  suffered  several  major  budget  cuts  dining  its  history. 
Interviews  with  two  previous  LMDSS  program  managers  revealed  estimates  of  the  time 
required,  man-hours,  and  size  of  LMDSS  related  work  were  largely  based  on  available 
funding  instead  of  requirements.  No  evidence  could  be  found  suggesting  that  automated 
software  tools.  Computer  Aided  Software  Engineering  (CASE)  tools,  or  standard 
software  metrics  were  used  to  develop  accurate  estimates  in  planning  and  scheduling 
LMDSS  work.  As  deadlines  passed  without  LMDSS  developers  meeting  stated  goals. 


101 


credibility  was  lost  with  upper  NAVAIR  management.  This  loss  of  credibility  negatively 
impacted  the  funding  support  received  by  the  LMDSS  program. 

d)  Proper  Contracting  Support 

Interviews  revealed  Small  Business  Administration  (SBA)  8A  contractors 
were  used  to  satisfy  key  requirements  during  the  transition  of  LMDSS  to  the  Web 
environment.  On  several  occasions,  the  primary  8A  contractor  failed  to  deliver  the 
desired  software  products  citing  misunderstandings  involving  LMDSS  requirements. 
More  detailed  requirements  were  drafted  by  the  LMDSS  program  manager  and  accepted 
by  the  contractor.  Subsequent  attempts  by  the  contractor  to  meet  these  detailed 
requirements  were  not  successful.  The  primary  LMDSS  8A  contractor  eventually  was 
forced  to  go  out  of  business  and  was  replaced  by  a  more  qualified  contractor. 

e)  New  Technology 

The  new,  complex  technologies  involved  in  building  a  Web  interface  to  a 
massive,  integrated  database  structure  has  overwhelmed  previous  LMDSS  development 
teams.  Additional  funding  to  support  developer  training  is  one  solution  to  dealing  with 
rapidly  changing  technology.  A  second  solution  is  to  outsource  using  skilled  contractors 
with  a  proven  track  record  of  successfully  completing  such  large-scale  database  projects. 
Either  alternative  will  require  additional  funding  commitments  by  NAVAIR. 

J)  Lack  of  User  Input 

The  literature  review  showed  how  the  lack  of  user  input  and  feedback  is  a 
common  problem  among  failed  software  development  projects.  Strategies  implementing 
user  focus  groups,  user  meetings,  and  periodic  e-mail  newsletters  are  all  effective  means 
of  correcting  this  problem.  Getting  back  in  touch  with  users  should  begin  with  an 
aggressive  attempt  to  update  the  entire  LMDSS  user  database  maintained  by  NAVAIR. 
Multiple  attempts  by  researchers  to  reach  a  random  sample  of  LMDSS  users  by  phone 
and/or  e-mail  resulted  in  successfully  contacting  74  percent  of  the  sample.  The  vast 
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majority  of  the  sampled  users  had  changed  e-mail  addresses  since  registering  with 
NAVAIR.  Correcting  the  e-mail  addresses  of  the  remaining  LMDSS  users  will  require  a 
dedicated  effort  but  will  be  worth  the  effort  if  a  LMDSS  e-mail  newsletter  is  ever 
implemented. 


g)  Personnel  Turnover 

LMDSS  has  had  more  than  its  share  of  personnel  turnovers  with  respect  to 
both  program  managers  and  programmers  over  its  eight-year  history.  Interview  findings 
identified  funding  cuts  as  a  major  cause  of  the  downsizing  and  personnel  turnovers 
suffered  by  the  LMDSS  program.  Three  NAVAIR  managers  interviewed  stated  these 
turnovers  had  a  negative  effect  on  the  LMDSS  project.  These  frequent  turnovers  also 
made  it  difficult  to  document  a  detailed  history  of  the  LMDSS  project. 

2.  Does  LMDSS  meet  user  requirements? 

During  the  majority  of  this  research  period,  LMDSS  was  involved  in  a  major 
overhaul.  The  survey  responses  and  interviews  confirmed  that  before  LMDSS  was  taken 
offline  in  December  1998,  it  did  not  provide  timely,  accurate  data  to  its  users. 

Developers  have  considered  LMDSS  to  be  in  an  evolutionary,  prototype  stage  for 
several  years  with  since  its  transition  to  the  Web  in  1994.  LMDSS  users  were  not  clearly 
informed  of  its  prototype  status  and  were  not  notified  of  the  data  error/corruption 
problems.  Moore  and  Snyder  (1998)  also  identified  this  lack  of  communication  between 
NAVAIR  and  LMDSS  users  during  their  research. 

The  limited  LMDSS  data  loads  (which  included  only  a  few  of  the  Navy's  many 
aircraft  type,  models,  and  series)  combined  with  frequent  errors  within  the  data  itself 
both  confused  and  frustrated  users.  The  majority  of  the  random  sample  of  queried 
LMDSS  users  abandoned  the  application  after  a  few  failed  attempts  to  acquire  useful 
data.  Despite  this,  user  interviews  revealed  a  common  eagerness  to  know  the  status  of 
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LMDSS,  if  it  was  finally  working,  and,  if  so,  whom  they  needed  to  contact  to  get  access 
again. 

Until  LMDSS  can  provide  access  to  timely,  accurate  AY-3M  data  relating  to  the 
Navy’s  entire  aircraft  inventory,  LMDSS  will  continue  to  be  of  little  value  to  its  users.  In 
addition  to  the  conversion  of  all  AV-3M  data  onto  the  new  Y2K  compliant  server, 
developers  are  currently  struggling  to  implement  a  data  warehouse  structure  where  all 
aviation  maintenance  and  logistics  databases  are  integrated,  compatible,  and  easily 
accessible  through  LMDSS  and  other  NALDA II  applications. 

3.  What  are  the  significant  decisions  users  would  like  LMDSS  to 
support? 

The  vast  majority  of  current  LMDSS  users  are  data  analysts  assigned  to  the 
various  organizations  listed  in  table  7.  Survey  responses  and  interviews  revealed  LMDSS 
is  used  more  as  a  database  interface  and  report  generator  than  as  a  decision  support  tool. 
Questions  in  the  Web  survey  designed  to  capture  the  significant  decisions  LMDSS  needs 
to  support  were  difficult  for  users  to  answer.  Frequently,  users  answered  with  requests 
for  specific  types  of  data  instead  of  addressing  the  types  of  decisions  such  data  need  to 
support.  This  was  particularly  apparent  in  the  survey  findings  from  questions  1 8  and  37. 

Interviews  revealed  major  decisions  supported  by  LMDSS  users  involve: 

•  Life-cycle  costs 

•  Reliability  centered  maintenance  decisions 

•  Historical  performance  of  aircraft  systems,  parts  and  engines 

•  The  impact  of  program  decisions  on  readiness 

•  Comparing  the  performance  of  maintenance  activities  at  all  levels 

•  Forecasts  of  the  future  reliability  of  aircraft  systems 
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PMAs  and  APMLs,  who  for  the  most  part  are  not  direct  users  of  LMDSS,  need  to 
be  queried  more  thoroughly  to  gain  sorely  needed  insight  into  what  specific  decisions 
need  to  be  supported  on  a  continual  basis. 


4.  What  are  the  key  attitudes  and  opinions  of  the  LMDSS  users? 

The  statistical  ANOVA  testing  conducted  on  the  four  group  comparison  questions 
described  in  methodology,  did  not  result  in  a  statistically  significant  difference  among  the 
various  user  groups  with  respect  to: 

•  User  requirements 

•  Modeling  and  simulation 

•  Graphics 

•  Exporting  LMDSS  data 

•  Ad-hoc  query  tool 

•  How  often  LMDSS  is  accessed 


Some  of  the  key  attitudes  expressed  by  LMDSS  users  during  this  research  were: 

•  Users  placed  a  strong  emphasis  on  receiving  accurate  and  timely  data. 

•  Users  have  been  frustrated  with  the  limited  data  and  quality  of  data 
available  through  LMDSS.  Users  have  abandoned  LMDSS  until  it 
delivers  the  data  they  need. 

•  Users  want  to  be  able  to  conduct  their  own  ad-hoc  queries  within  LMDSS. 
Most  feel  that  additional  IQ  training  is  required. 

•  Users  want  to  be  able  to  export  data  downloaded  from  LMDSS  into  a 
familiar  user  tool  like  Excel  so  that  they  can  do  graphical  presentations. 
An  internal  graphical  capability  would  be  welcomed  but  the  primary 
concerns  were  exportation  of  data  and  ease  of  use. 
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•  Based  upon  their  past  experiences,  users  do  not  trust  LMDSS  as  a  reliable 
source  of  data  and  therefore  want  to  know  how  data  is  derived  and 
defined. 

•  Users  overwhelmingly  agreed  that  e-mail  is  the  best  way  for  NAVAIR  to 
stay  in  touch  with  its  customers  at  all  levels. 

5.  How  has  LMDSS  been  funded? 

In  the  past,  LMDSS  has  received  its  funds  from  the  NALDA  program  funding  and 
has  never  been  assigned  its  own  line  of  funding.  LMDSS  managers  identified  severe  cuts 
in  funding  and  personnel  as  the  major  causes  for  many  of  the  delays  and  problems 
suffered  during  the  eight-year  LMDSS  development  effort.  Recently,  much  of  the 
funding  for  LMDSS  development  efforts  has  come  "out  of  hide"  from  NAVAIR's 
Operation  and  Maintenance  funds. 

As  NAT  DA  I  is  closed  down  in  October  1999,  LMDSS  will  take  on  a  far  greater 
importance  as  the  primary  gateway  to  all  of  the  Navy's  AV-3M  historical  data.  An 
aggressive  POM  effort  needs  to  be,  initiated  to  acquire  the  funding  necessary  to  finish 
existing  plans  for  LMDSS  and  to  ensure  proper  funding  for  maintenance  and  future 
development 

6.  How  can  LMDSS  be  improved  to  better  meet  managerial  needs? 
a)  Provide  accurate  and  timely  data. 

A  common  perception  held  by  LMDSS  users  is  that  most  AV-3M  data 
errors  occur  at  the  hangar  deck  level  because  maintenance  technicians  do  not  have  an 
appreciation  of  how  important  it  is  to  provide  correct  maintenance  and  logistics  data.  An 
effort  to  work  with  the  Chief  of  Naval  Education  and  Training  (CNET)  to  include  more 
detailed  NALDA/LMDSS  training  in  the  appropriate  A  schools  and  technical  training 
centers  would  go  far  to  help  correct  this  data  entry  problem.  A  CD-ROM  multimedia 
tutorial  mailed  to  all  Fleet  aviation  commands  might  be  one  way  to  alleviate  this 
problem.  Such  a  tutorial  could  show  maintenance  and  logistics  technicians  how 
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important  AV-3M  data  is  to  making  correct  strategic  program  decisions  at  the  PMA 
level. 

An  aggressive  effort  by  NAVADR.  is  needed  to  hold  wings  and  squadrons 
accountable  for  the  quality  of  AV-3M  data  submitted.  Linking  this  effort  to  periodic 
command  inspections  would  do  much  to  motivate  squadrons  to  pay  closer  attention  to  the 
quality  of  their  submitted  data. 

b)  Make  LMDSS  more  user  friendly. 

Although  a  monumental  effort  has  been  made  to  make  LMDSS  easier  to 
use,  only  twenty-six  percent  of  the  users  who  responded  to  the  Web  survey  considered 
LMDSS  to  be  user-friendly.  The  poor  quality  of  LMDSS  data  and  the  significant  effort 
involved  in  accessing  needed  data  surely  contributed  to  this  attitude. 

An  idea  promoted  by  the  current  NAVAIR  3.6.2  would  be  to  customize 
LMDSS  for  each  individual  user  in  a  manner  similar  to  the  Yahoo  and  CNN  Web  sites. 
Upon  first  registering  at  the  LMDSS  site,  a  user  could  customize  the  site  to  provide  only 
specifically  required  data  and  reports.  This  personalized  site  environment  would  appear 
every  time  the  user  subsequently  logged-on  to  LMDSS.  All  desired  data  and  reports 
would  be  updated  and  available  every  time  the  user  visited  his  uniquely  designed  LMDSS 
site.  Automated  tools  could  be  used  to  inform  the  user  when  requested  metrics  of  interest 
had  changed.  Current  Web  technology  that  pushes  data  to  personalized  Web  sites  could 
be  implemented  to  continually  update  a  user's  individualized  "My  LMDSS  Home  Page" 
with  the  most  current  data  available.  Personalized  icons  could  be  created  by  the  user  on 
this  home  page  to  quickly  bring  up  commonly  needed  reports  containing  the  most  recent 
data. 

A  file  folder  tab  format  as  found  on  many  of  today's  popular  Web  sites 
might  help  LMDSS  users  find  their  way  to  desired  data  easier  than  requiring  them  to  drill 
down  into  imbedded  modules.  Interviews  revealed  users  want  more  extensive  help 
screens  and  help  functions.  Users  place  a  higher  importance  on  just  getting  accurate. 
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timely  data  than  details  concerning  report  formats.  As  one  analyst  stated,  "Using  LMDSS 
needs  to  be  as  easy  as  following  the  bouncing  ball." 

c)  Enable  users  to  easily  export  LMDSS  data  to  an  application 
supporting  graphics  and  spreadsheets  like  MS  Excel 

LMDSS  users  strongly  expressed  their  need  to  cleanly  export  data  to 
applications  with  graphics  capabilities  like  Excel.  The  survey  questions  relating  to  this 
topic  received  the  most  responses  in  both  quantity  and  in  the  strength  of  the  users' 
conviction.  (See  results  of  questions  24  and  32  in  the  Results  section  of  Findings.) 
Although  internal  graphical  capabilities  were  also  desired,  the  users  were  clear  that  they 
needed  to  be  able  to  cleanly  export  data  to  complete  further  analysis  on  the  material 
provided  by  LMDSS.  Users  interviewed  seemed  willing  to  make  compromises  on 
specific  data  format  as  long  as  the  data  was  accurate,  timely,  and  could  be  downloaded  to 
an  application  and  manipulated  by  the  user. 

d)  Revise  the  IQ  tool  to  be  more  user  friendly  or  abandon  the 
current  IQ  tool  and  develop  a  new,  easier  to  use,  ad-hoc  query 
tool  within  LMDSS. 

LMDSS  users  place  a  high  importance  on  their  ability  to  develop 
individualized  ad-hoc  queries.  Users  familiar  with  the  IQ  query  tool  within  LMDSS 
reported  it  as  too  difficult  to  use.  Only  two  percent  of  LMDSS  users  surveyed  found  the 
IQ  tool  to  be  user  friendly.  Users  expressed  their  frustration  upon  completing  IQ  tool 
training  when  they  discovered  they  were  unable  to  overcome  LMDSS  access  and 
software  difficulties.  Others  reported  losing  most  of  the  IQ  skills  acquired  during 
training  when  they  were  unable  to  practice  using  the  tool  because  of  various  problems 
accessing  LMDSS  data. 

A  search  engine  similar  to  those  found  in  other  Web  sites  (altavista.com, 
dogpile.com,  yahoo.com,  amazon.com)  is  one  suggestion  for  making  ad-hoc  queries 
easier  to  implement.  Data  contained  in  common  report  templates  could  be  developed 
with  the  help  of  extensive  user  interviews.  These  reports  could  be  easily  accessed  by  an 
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Internet  style  search  engine  interface  familiar  to  most  Web  surfers.  Such  a  search  engine 
would  make  it  easier  for  users  to  conduct  a  customized  search.  The  search  engine  would 
hide  the  complicated  details  of  SQL  coding  from  the  user  just  as  Web  site  authoring  tools 
now  hide  the  complex  details  of  HTML  code. 

e)  Provide  a  modeling/simulation  capability. 

If  LMDSS  is  ever  upgraded  to  become  a  fully  functioning  decision  support 
tool,  as  its  name  implies,  it  will  have  to  eventually  include  extensive  modeling  and 
simulation  capabilities.  As  shown  in  the  literature  review,  modeling  and  simulations  are 
an  essential  part  of  a  DSS.  Previous  LMDSS  research  by  Moore  and  Snyder  (1998) 
determined  modeling  is  needed  by  both  PMAs  and  APMLs  to  better  forecast  life-cycle 
costs  and  predict  the  effect  of  program  decisions  on  those  costs  and  overall  readiness. 

The  active  involvement  of  PMA/APMLs  will  be  required  to  develop  the 
models  and  simulations  needed  to  meet  their  unique  decision  support  needs.  DSS 
modeling  requirements  will  have  to  be  thoroughly  documented  and  financially  supported 
in  the  Program  Objectives  Memorandum  process  by  NAVAER  senior  leadership  to  be 
successfully  implemented. 

Some  of  the  modeling  applications  LMDSS  users  suggested  included 
models  that  would: 

•  Assist  in  predicting  future  maintenance  support  requirements 

•  Estimate  life  cycle  costs  and  the  impacts  of  aging 

•  Capture  various  return  on  investments  for  proposed  readiness  and  program 

changes 

•  Forecast  spare  costs  and  failure  trends 

A  prototype  model  built  by  the  authors  of  this  thesis  provides  an  example  of  how 
modeling  and  simulation  could  assist  logistics  and  maintenance  managers.  The  model, 
designed  to  help  wing  maintenance  managers  forecast  the  readiness  impact  of  a  real- 
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world  depot  support  problem,  is  presented  in  Appendix  F.  The  model  enables  users  to 
conduct  sensitivity  and  "what  if'  analysis  to  assist  in  making  strategic,  unstructured 
decisions. 


7.  How  can  NAVAIR  improve  the  LMDSS  program? 

NAVAIR  has  several  major  difficulties  facing  them  as  they  struggle  to  make 
LMDSS  an  effective  analysis  tool  for  the  Fleet.  These  include: 

•  Ensuring  the  final  LMDSS  product  performs  properly  within  established 
quality  specifications. 

•  Ensuring  LMDSS  meets  user  requirements. 

•  Convincing  users  LMDSS  is  capable  of  providing  accurate,  meaningful 
data. 

•  Training  new  and  repeat  users  how  best  to  use  LMDSS. 

•  Continuing  to  develop  new  and/or  better  capabilities  into  the  system. 

Although  the  findings  of  this  research  discovered  numerous  areas  that  could  be 

improved  upon,  the  following  are  the  significant  areas  of  concern  identified  by  the 
research: 

a)  Improve  communications  with  users 

Managers  interviewed  within  the  LMDSS  development  team  believed  the 
experts  on  the  QA  team  and  others  at  NAVAIR  knew  what  the  LMDSS  users  wanted 
without  asking  the  users  themselves.  NALDA  user  meetings  were  abandoned  as  NALDA 
II  and  LMDSS  underwent  development  with  the  intention  of  reinstating  them  in  the 
summer  of  1998.  Months  turned  into  years  without  a  NALDA  or  LMDSS  user  meeting. 
Over  eighty  percent  of  the  users  participating  in  the  Web  survey  reported  they  had  not 
been  contacted  for  input  by  the  LMDSS  development  team.  Only  three  percent  reported 
attending  NAVAIR  sponsored  LMDSS  user  group  meetings.  Given  the  project’s  history 
of  missing  deadlines  and  not  meeting  advertised  expectations,  a  dedicated  effort  needs  to 
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be  made  to  improve  communications  between  the  NAVAIR  developers  and  the  Fleet 
customers.  This  includes  identifying  a  timely  means  to  provide  information  to  the  users 
on  the  status  of  LMDSS  and  a  means  of  obtaining  feedback  from  the  users.  Elimination 
of  miscommunication  will  go  a  long  way  toward  reestablishing  a  favorable  reputation  of 
LMDSS  within  the  user  ranks. 

Teleconferencing,  focus  groups,  and  regional  user  meetings  are  alternative 
means  to  continuing  the  critical  effort  of  staying  in  touch  with  LMDSS  customers  in  a 
fiscally  constrained  environment.  Accurate  user  feedback  and  input  is  critical  to 
maintaining  meaningful  LMDSS  software  requirements.  Assuming  those  on  the  LMDSS 
development  team  already  know  what  LMDSS  users  want  and  need  is  a  risky 
assumption. 

Survey  respondents  suggested  NAVAIR  stay  in  touch  with  users  through 
the  use  of  an  email  LMDSS  newsletter.  Since  99  percent  of  the  users  within  the  LMDSS 
random  sample  contacted  during  this  research  have  email  accounts  one  can  assume  the 
majority  of  the  total  LMDSS  user  population  also  have  email  accounts.  Only  two  users 
contacted  were  aware  of  the  data  problems  within  LMDSS  that  resulted  in  data  being 
corrupted  and  frozen  for  several  months  in  the  fall  of  1998.  During  this  period,  users 
were  still  going  to  the  LMDSS  Web  site  for  periodic  data  searches.  A  NAVAIR 
sponsored  LMDSS  e-mail  newsletter  would  help  avoid  such  misunderstandings,  maintain 
the  trust  and  cooperation  of  users,  and  assist  greatly  in  collecting  user  feedback. 

b)  Improve  LMDSS  training. 

User  comments  emphasized  the  importance  of  hands-on  training  with 
students  being  assigned  to  their  own  computer  terminal  and  practicing  with  a  more 
realistic  sample  database.  Users  expressed  their  frustration  of  completing  LMDSS 
training  only  to  discover  LMDSS  was  still  under  development,  delivered  inaccurate  data, 
and  did  not  work  as  promised. 

To  avoid  these  misunderstandings,  further  training  should  be  delayed  until 
LMDSS  is  fully  operational.  The  abundance  of  non-response  answers  to  survey  questions 
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and  the  interview  results  reveal  users  have  abandoned  LMDSS  and  will  continue  to  do  so 
until  it  is  fixed.  Continuing  to  conduct  training  before  LMDSS  can  deliver  as  promised 
will  only  worsen  NAVAIR's  relationship  with  its  customers. 

CNET's  Electronic  Schoolhouse  Network  (CESN)  should  be  explored  to 
expand  the  scope  of  LMDSS  training  via  distance  learning.  CNET  has  built  several 
electronic  classrooms  that  are  available  to  support  LMDSS  training.  All  of  the  CESN 
sites  are  located  at  the  major  Fleet  concentration  areas.  In  1997,  a  CNET  pilot  project 
was  conducted  to  test  the  viability  of  completing  hands  on  software  training  via  distance 
leaning.  Software  training  originating  at  the  Naval  Reserve  Professional  Development 
Center  in  New  Orleans  was  transmitted  to  remotely  located  electronic  classrooms 
operated  by  CESN.  Software  interface  difficulties  limited  the  success  of  this  pilot 
project,  but  CNET  has  continued  to  work  toward  improving  the  viability  of  distance 
learning  and  is  working  on  attaining  the  capability  for  CESN  to  conduct  remote  software 
training.  Utilizing  CESN  sites  would  also  allow  NAVAIR  to  conduct  timely,  periodic 
LMDSS  focus  groups  and  user  feedback  meetings  while  avoiding  cost  prohibitive  travel 
expenses. 
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VIL  CONCLUSIONS/RECOMMENDATIONS 


A.  THE  NEED  FOR  A  STRATEGIC  DECISION  SUPPORT  SYSTEM 

1.  Conclusion:  LMDSS  does  not  meet  the  original  requirements 

established  for  a  strategic  decision  support  system  for  PMAs/APMLs. 

The  original  requirement  for  LMDSS  was  established  by  NAVAIR  04  and  the 
former  Aviation  Supply  Office  (currently  renamed  NAVICP)  in  June  1991.  This 
requirement  directed  the  development  of  a  strategic  decision  support  system  to  assist 
PMA,  APML,  and  WSM  teams  in  measurably  reducing  program  life  cycle  costs  while 
minimizing  negative  effects  on  Fleet  readiness.  Many  specific  DSS  capabilities  were 
initially  promised  to  the  NAVAIR  chain  of  command  in  1993.  It  was  believed  the 
developed  program  LMDSS  would: 

•  Provide  a  single  source  of  real-time  maintenance  and  logistics  data  by 
accessing  as  many  as  nine  stovepipe  databases 

•  Allow  sensitivity  or  “what  if’  analysis  based  on  the  ability  to  manipulate 
supply  and  maintenance  parameters 

•  Support  detailed  causative  research 

•  Provide  an  audit  trail  of  a  manager's  decision  process 

•  Forecast  engine  problems  and  repair  alternatives  using  the  corporate 
expert  knowledge  of  top  engine  analysts 

•  Assess  the  repair  productivity  and  efficiency  of  maintenance  activities 
across  all  organizational  levels 

•  Deliver  presentation  quality  graphic  (Naval  Air  Systems  Command, 
LMDSS  Requirements  Document,  1993) 

The  current  operational  version  of  LMDSS  has  none  of  these  capabilities.  During 
its  eight-year  development,  LMDSS  has  evolved  from  a  DSS  design  for  PMs  and  APMLs 
into  a  limited  management  information  system  (MIS)  used  by  data  analysts  at  all  levels. 
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Recommendation:  Rename  LMDSS  and  call  it  a  management 
information  system.  Validate  the  requirement  for  a  strategic  decision 
support  system  to  help  PMAs,  APMLs,  and  WSMs  better  manage 
their  programs. 

LMDSS  users  contacted  during  this  research  expressed  their  loss  of  faith  in 
LMDSS'  capabilities  as  the  primary  reason  for  abandoning  it.  Moore  and  Snyder  (1998) 
concluded  LMDSS  did  not  meet  the  criteria  for  a  DSS.  Consideration  should  be  given  to 
changing  the  name  of  LMDSS  to  more  accurately  reflect  its  role  as  a  Web  interface  that 
provides  AV-3M  data  for  Fleet  and  civilian  data  analysts.  The  capability  of  LMDSS  to 
drill  down  into  AV-3M  data,  identify  primary  cost  drivers,  present  standardized  reports, 
and  support  ad-hoc  queries  places  it  in  the  management  information  system  category. 
When  LMDSS  becomes  fully  operational  a  new  name  might  help  restore  user  faith  and 
avoid  misunderstandings.  A  more  accurate  name  for  LMDSS  would  be  the  Logistics 
Management  Information  System  (LMIS). 

The  original  requirement  for  a  strategic  decision  support  system  to  improve 
program  management  needs  to  be  further  explored  and  validated  again.  If  this 
requirement  is  determined  to  still  be  valid,  we  recommend  a  strategic  decision  support 
system  containing  models  and  simulations  and  supported  by  an  integrated  data 
warehouse  be  built  for  NAVAIR  program  managers.  Such  a  system  could  provide 
invaluable  assistance  to  PMAs,  APMLs,  and  WSMs  in  their  daily  struggle  with 
unstructured  decisions  to  overcome  limited  funding  and  maintain  readiness. 

2.  Conclusion:  The  key  to  providing  accurate,  timely  logistics  and 
management  data  lies  in  the  development  of  a  data  warehouse. 

An  effective  strategic  decision  support  system  for  NAVAIR  managers  will  require 
the  integration  of  data  currently  found  in  stovepipe,  legacy  and  relational  databases 
within  NALDA  I.  A  three-tier,  data  warehouse  architecture,  containing  a  meta-data 
capability,  with  a  middleware  based,  application  interface  holds  the  best  promise  for 
achieving  this  integration. 
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Recommendation:  Develop  a  data  warehouse  to  support  LMDSS. 

Developing  a  data  warehouse  that  seamlessly  integrates  numerous  NAVAIR 
legacy  databases  to  provide  easy  access  to  timely,  accurate  data  is  a  major  undertaking 
for  any  organization.  A  detailed,  thorough  effort  to  benchmark  other  organizations  that 
have  successfully  completed  such  a  project  should  yield  invaluable  "lessons-leamed." 

To  build  a  data  warehouse,  transformation  and  extraction  tools  will  be  needed  to 
access  the  various  data  types.  Meta-data  managers  will  be  needed  to  describe  and  track 
the  origins  of  data,  how  it  was  cleaned  and  distributed,  who  is  responsible  for  it,  and  how 
frequently  it  is  updated.  In  addition,  business  intelligence  tools  and  data  cleansing  tools 
will  also  be  required.  (Malloy,  1997) 

B.  SOFTWARE  DEVELOPMENT  ISSUES 

1.  Conclusion:  The  absence  of  a  formal  risk  management  program  is  a 
major  contributor  to  the  failure  of  the  eight-year  LMDSS  program. 

A  properly  implemented,  formal  risk  management  program  is  a  proven  software 

development  tool  for  identifying,  evaluating  and  mitigating  potential  risks.  The  literature 

review  chapter  describes  research  showing  the  impact  an  effective  risk  management 

program  can  have  on  the  success  of  a  software  development  effort.  Although  interviews 

led  us  to  believe  formal  risk  management  practices  are  used  in  other  areas  of  the  NALDA 

program,  the  LMDSS  project  does  not  appear  to  have  a  risk  management  policy  in  effect. 

Recommendation:  Institute  a  formal  LMDSS  risk  management 
program 

Formal  policies  requiring  periodic  reviews  of  existing  and  emerging  risks  present 
a  proven  method  for  software  managers  to  manage  the  potential  risks  to  a  software 
development  project.  Risk  management  will  allow  the  LMDSS  development  team  to 
anticipate  obstacles  and  proactively  develop  strategies  to  mitigate  them. 

Although  a  variety  of  formal  methods  exist,  developing  and  tracking  an  up  to  date 
list  of  "Top  Ten  Risks"  to  the  LMDSS  program  is  a  recommended,  proven  strategy  of 
communicating  potential  obstacles  to  both  managers  and  developers  (McConnell,  1996). 
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As  risks  are  controlled  and  minimized  they  can  be  removed  from  the  list  and  replaced  by 
higher  priority  risks. 

2.  Conclusion:  Documentation  in  all  areas  of  the  LMDSS  program  needs 
to  be  improved. 

As  stated  in  the  discussion  section,  no  evidence  could  be  found  suggesting  the 

LMDSS  project  used  automated  software  tools,  CASE  tools  or  standardized  software 

metrics  to  develop  accurate  estimates  for  planning  and  workload  scheduling.  Without 

accurate,  credible  estimates  of  man-hours  required,  anticipated  lines  of  code  and  other 

proven  software  development  metrics,  schedules  and  deadlines  became  best  guesses. 

The  inability  to  meet  poorly  estimated  deadlines  put  additional  unneeded  pressure  on 

LMDSS  team  members.  Poor  documentation  also  made  it  difficult  for  LMDSS  program 

managers  to  fight  for  limited  funds.  As  a  result,  the  available  funding  drove  LMDSS 

requirements  instead  of  the  requirements  driving  the  level  of  funding.  This  resulted  in  a 

“death  spiral”  causing  further  delays  and  decreased  capabilities. 

Recommendation:  Improve  the  documentation  maintained  by  the 
LMDSS  software  development  team. 

In  the  same  way  that  trend  analysis  and  configuration  management  are 
instrumental  in  improving  aviation  maintenance  practices,  documentation  is  just  as 
important  to  software  development.  Metrics  need  to  be  identified  and  tracked  in  order  to 
measure  progress  (or  lack  of  progress)  in  the  LMDSS  program.  Proper  metrics  are 
imperative  in  order  to  implement  any  quality  assurance  programs.  Quality  cannot  be 
improved  is  there  are  not  any  standardized  methods  with  which  to  identify  performance 
or  process  problems.  Risk  management  documentation  and  meticulous,  thorough 
schedules  based  on  accurate  estimates  from  software  metrics  can  be  influential  tools  in 
future  attempts  to  attain  additional  LMDSS  funding  support  from  upper  management. 

CASE  tools  should  be  reviewed  for  applicability  and  implemented  where 
appropriate.  A  dedicated  effort  needs  to  be  made  to  document  program  metrics  such  as 
man-hours  expended,  Lines  of  Code  (LOC)  completed,  and  software  errors  per  LOC  (bug 
density).  Contractors  should  also  be  providing  similar  metric  data  on  their  development 
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process  for  analysis.  This  data  should  provide  managers  with  a  better  means  on  how  best 
to  allocate  funding  and  manpower  resources  to  meet  cost  and  time  constraints.  Data 
sampling  is  another  method  for  verifying  accuracy.  In  addition,  accurate  documentation 
is  crucial  to  conducting  thorough  post-mortem  (or  lessons  learned)  summaries  to  prevent 
repetition  of  process  problems  in  future  software  projects. 

3.  Conclusion:  Major  LMDSS  requirement  changes  significantly 
contributed  to  its  delayed  deployment. 

LMDSS  managers  continue  to  propose  major  requirement  changes  prior  to 

successfully  completing  the  current  phase  of  development.  As  shown  consistently  in  the 

software  development  literature,  the  delays  and  costs  associated  with  changing  core 

requirements  late  in  the  developmental  cycle  is  exponentially  proportional  to  the  length 

of  time  in  the  cycle.  More  simply  stated,  the  earlier  a  change  is  implemented,  the  less 

impact  it  will  have  on  schedule  and  cost.  When  changes  are  made  in  the  middle  or  late  in 

a  SW  development  project,  it  does  not  matter  whether  the  reason  is  due  to  changes  in 

funding,  technology  and  personnel.  The  result  will  be  the  same:  drastic  schedule  delays 

and/or  exorbitant  cost  increases.  LMDSS  suffered  every  possible  requirement  change 

including  hardware  requirements,  software  language,  contractor  changes,  personnel 

changes  and  user  requirement  changes.  Given  this,  it  is  no  wonder  that  LMDSS  is  over 

budget,  behind  schedule  and  not  fully  functional. 

Recommendation:  Identify  and  validate  detailed  requirements  for 
LMDSS  and  implement  a  versioning  strategy  to  manage  future 
requirements  changes. 

LMDSS  has  gone  through  so  many  metamorphic  changes  users  do  not  have  a 
clear  understanding  of  its  current  capabilities.  Program  managers  need  to  get  back  in 
touch  with  LMDSS  users  and  thoroughly  document  their  requirements.  When  LMDSS  is 
fully  deployed,  NAVAIR  needs  to  properly  advertise  its  strengths  and  limitations  to  its 
customers.  As  shown  in  the  findings,  the  majority  of  users  did  not  know  specifically 
what  LMDSS  could  do. 
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Implementing  a  versioning  technique,  similar  to  Microsoft’s  tactic  of  developing 
its  MS  Office  line,  would  enable  continued  developmental  improvements  while  still 
focusing  programmers  on  meeting  core  program  requirements.  LMDSS  needs  to  adopt 
a  policy  that  mirrors  Microsoft’s  Office  marketing  practices.  Program  managers  need  to 
identify  a  core  set  of  capabilities  that  LMDSS  should  have  and  strive  to  complete 
production.  Major  changes  resulting  from  new  technologies,  new  personnel  or  funding 
constraints  should  be  reviewed  for  potential  integration  into  LMDSS.  If  evaluations  of 
new  requirements  are  favorable,  a  plan  should  be  developed  to  implement  the  new 
requirements  into  a  future  version  of  the  application  instead  of  trying  to  continually 
change  the  existing  development  process. 

4.  Conclusion:  Lack  of  continued  user  involvement  in  the  LMDSS 

development  project  contributed  to  its  failure  to  meet  customer  needs. 

As  covered  extensively  in  the  Findings  chapter  and  again  in  the  Discussion  and 

Analysis  chapter,  communications  between  users  and  developers  is  poor.  Over  eighty 

percent  of  the  users  participating  in  the  Web  survey  reported  they  had  not  been  contacted 

for  input  by  the  LMDSS  development  team  and  only  three  percent  reported  attending 

NAVAIR  sponsored  LMDSS  user  group  meetings.  Due  to  the  years  of  conflicting  reports 

about  the  capabilities  of  the  LMDSS  application,  most  users  do  not  have  a  clear 

understanding  of  what  exactly  the  LMDSS  program  can  do.  As  evidenced  in  the  survey 

results,  many  users  are  willing  to  provide  input  but  have  not  been  asked. 

Recommendation:  NAVAIR  needs  to  improve  communications  with 
LMDSS  users. 

Findings  showed  users  prefer  a  NAVAIR  sponsored  LMDSS  e-mail  newsletter 
but  teleconferencing,  focus  groups  and  regional  user  meetings  are  alternative  means  of 
improving  user  feedback  and  ensuring  two  way  communications. 

In  order  for  LMDSS  to  better  meet  user  requirements,  NAVAIR  managers  need  to 
actively  involve  users  in  the  development  process.  Periodic  LMDSS  user  meetings,  focus 
groups,  and  aggressive  e-mail  correspondence  are  recommended  strategies  for  NAVAIR 
to  employ  to  get  back  in  touch  with  its  LMDSS  customers. 
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5.  Conclusion:  Until  LMDSS  can  provide  access  to  all  AV-3M  data  for 
every  platform  in  the  Navy’s  current  inventory,  LMDSS  will  continue 
to  not  meet  user  needs. 

As  stated  before,  without  AV-3M  data  properly  loaded  for  all  type/model/series 
and  type  equipment  codes,  LMDSS  will  continue  to  provide  only  limited  usefulness. 
Interviews  with  NAVAIR  managers  revealed  LMDSS  developers  failed  to  appreciate  the 
complexity  of  building  an  integrated  database  support  structure.  Y2K  compliance 
measures,  inadequate  contracting  and  data  security  concerns  further  complicated  the 
transition  of  data  into  the  system.  This  has  resulted  in  repeated  difficulties  loading  the 
data. 

Recommendation:  Complete  the  loading  of  AV-3M  data  for  all 
type/model/  series  aircraft  to  enable  access  by  LMDSS  users. 

All  data  needs  to  be  loaded  as  soon  as  possible  so  that  all  users  may  obtain  the 
data  required.  Future  LMDSS  development  efforts  need  to  be  supported  by  adequate 
funding,  skilled  data  warehouse  experts,  and  a  major  contractor  with  a  track  record  of 
implementing  similar  data  warehouse  projects. 

C.  AREAS  FOR  FURTHER  RESEARCH 

The  following  areas  for  future  LMDSS  research  are: 

•  Conduct  a  follow-on  survey  of  LMDSS  users  once  LMDSS  is  fully 
functioning  with  access  to  all  historical  aircraft  and  logistic  data. 

•  Explore  the  NALDA  II  data  warehouse  plan  and  research  alternative 
strategies  to  support  LMDSS  integration. 

•  Conduct  an  in-depth  analysis  into  the  causes  of  LMDSS  data  errors  and 
recommend  ways  to  improve  the  quality  of  LMDSS  data  reports. 

•  Validate  and  define  the  need  for  a  strategic  DSS  by  NAVAIR  program 
managers. 

•  Develop  models  to  assist  NAVAIR  program  managers  in  making  strategic, 
unstructured  decisions  with  the  goal  of  improving  readiness  while  cutting 
program  costs. 
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APPENDIX  A.  CLASSIC  SOFTWARE  DEVELOPMENT  MISTAKES 


McConnell  (1996)  identified  and  described  thirty-six  classic  mistakes  that  directly 
lead  to  a  failed  software  development  effort.  His  list  is  divided  into  the  four  categories  of 
people,  process,  product,  and  technology. 

1.  People 

•  Undermined  motivation 

•  Weak  personnel 

•  Uncontrolled  problem  employees 

•  Heroics 

•  Adding  people  to  a  late  project 

•  Noisy,  crowded  offices 

•  Friction  between  developers  and  customers 

•  Unrealistic  expectations 

•  Lack  of  effective  project  sponsorship 

•  Lack  of  stakeholder  buy-in 

•  Lack  of  user  input 

•  Politics  placed  over  substance 

•  Wishful  thinking 

2.  Process 

•  Overly  optimistic  schedules 


121 


•  Insufficient  risk  management 

•  Contractor  failure 

•  Insufficient  planning 

•  Abandonment  of  planning  under  pressure 

•  Wasted  time  during  the  fuzzy  front  end 

•  Cuts  to  requirements  analysis,  architecture,  and  design 

•  Inadequate  design 

•  Shortchanged  quality  assurance 

•  Insufficient  management  controls 

•  Premature  or  overly  frequent  convergence 

•  Omitting  necessary  tasks  from  estimates 

•  Planning  to  catch  up  later 

•  Code-like-hell  programming 

3.  Product 

•  Requirements  gold-platting  (more  requirements  than  needed) 

•  Feature  creep  (changing  requirements) 

•  Developer  gold-platting 

•  Approval  of  a  schedule  slip  while  adding  additional  requirements 

•  Research-oriented  development 

4.  Technology 

•  Silver-bullet  syndrome  (a  single  new  practice  will  solve  schedule  problems) 
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•  Overestimated  savings  from  new  tools  or  methods 

•  Switching  tools  in  the  middle  of  a  project 

•  Lack  of  automated  source-code  control 
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APPENDIX  B.  NALDA  U 


The  Naval  Aviation  Logistics  Data  and  Analysis  (NALDA)  system  is  an 
automated,  relational  database  and  information  retrieval  system  for  aviation  logistics 
management  and  technical  decision  support.  NALDA  Phase  I  first  became  operational  in 
the  early  1980s  and  evolved  from  the  need  to  improve  the  data  analysis  capabilities  of 
logistics  and  aviation  program  managers  as  Navy's  aviation  weapons  and  support  systems 
became  more  complex  and  sophisticated.  In  the  mid  1990s,  NALDA  Phase  II  expanded 
the  data  base  and  analysis  capability  to  include  all  Fleet  logistics  elements.  (SPA WAR, 
1999) 

The  primary  objectives  of  the  NALDA  Phase  II  system  is  to  leverage  information 
technology  to  achieve  process  improvements  through  the  following: 

•  More  timely  and  accurate  information  and  tools 

•  Affordable  readiness  and  total  ownership  cost  metrics  and  process  tools  for 
decision  makers 

•  Easier  information  access  to  managers,  logisticians,  engineers  and  maintenance 
technicians  at  all  levels  within  the  Fleet 

•  Create  and  store  data  once,  and  use  it  many  times 

•  Integrate  with  the  Department  of  Defense's  Standard  and  open  systems 
architecture,  minimizing  stovepipes,  redundancies,  and  islands  of  automation. 
(NAVAIR,  1999) 

All  NALDA  II  applications,  one  of  which  is  LMDSS,  will  be  structured  within  a 
single  shared,  logical,  and  physically  distributed  Integrated  Weapons  Systems  Data  Base 
(IWSDB).  The  IWSDB  will  be  a  centralized  Oracle  object-oriented  data  base  structure 
that  will  contain  all  relevant  naval  aviation  logistic  data.  (Joseph  Interview,  1998) 
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APPENDIX  C.  FOCUS  GROUP  QUESTIONS 


Initial  questions: 

□  Who  are  the  Users?  List  Categories/Activities 

□  What  are  the  levels  of  users? 

□  Demographics?  Provide  information  about  the  users.  See  below. 

□  What  are  the  users  doing  with  LMDSS  now  from  your  perspective? 

□  What  are  the  most  common  help  requests? 

□  What  do  the  users  want  to  do  with  it  now? 

□  What  is  the  future  of  LMDSSS  going  to  look  like? 

□  What  do  you  like/dislike  about  the  old  survey? 

Additional  questions  that  arose  during  the  focus  group 

□  Do  TYCOMS  hire  people  that  may  also  have/need  access? 

□  Do  contractors  have  restricted  access? 

□  What  information  do  you  want  to  receive  from  the  survey? 

□  What  about  the  change  from  Unix  to  the  Web? 

□  Do  you  think  the  lost  graphical  capabilities  will  be  a  problem? 

Demographics  recommended  by  focus  group: 

□  Command/Activity 

□  Type  Users 

□  Active  Duty 

□  Contractor 

□  Civilian 

□  GS  Worker 

□  Job  Title 

□  IT  Experience 

□  Rate/Rank 

□  Internet  access 

□  Frequency 

□  Type  connection  -  modem/network  connection 

□  Type  organizational  category 

□  AIMD 

□  APML 

□  Staff 

□  Depot 
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APPENDIX  D.  WEB-BASED  SURVEY 


Logistics  Management  Decision  Support  System  (LMDSS) 

Survey 

Thank  you  for  your  assistance  in  this  LMDSS  research  effort  sponsored  by  NAVAIR 
3.6.2.  This  survey  was  prepared  by  CDR  Mark  Krause  (mwkrause@mbay.net)  and 
LCDR  Ellen  Evanoff  (evanofffus@aol.com)  from  the  Naval  Postgraduate  School.  The 
purpose  of  our  research  is  to  assist  NAVAIR  with  the  identification  of  LMDSS  user 
requirements  in  order  to  develop  a  better  decision-making  tool  for  the  Logistics  and 
Maintenance  communities.  The  quality  of  the  data  from  this  survey  will  depend  solely 
on  the  honesty  and  accuracy  of  your  input. 

In  Feb  99,  LMDSS  was  removed  from  the  Web  and  is  currently  being  reworked  and 
updated.  Despite  the  fact  LMDSS  will  not  be  available  for  use  for  several  months,  we 
are  very  much  interested  in  your  past  experience  with  LMDSS  and  what  you  would  like  it 
to  do  for  you  in  the  future. 

The  identity  of  those  responding  to  this  survey  will  be  kept  strictly  confidential.  Some 
basic  information  is  requested  in  case  we  need  to  contact  you  at  a  later  date.  If  there  are 
any  questions,  both  CDR  Krause  and  LCDR  Evanoff  can  be  reached  via  their  email 
addresses  listed  above.  So,  lets  get  started! 


Q.Name 

What  is  your  name? 


Q.  Email 

What  is  an  email  address  we  may  use  to  contact  you? 


Q.  Command 

Please  provide  your  command  name  (or  name  of  business)  and  job  title. 
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1 .  How  often  did  you  access  LMDSS  when  it  was  available  on  the  Web? 
O  Infrequently 

O  Monthly 
O  Weekly 
O  Daily 
O  Don't  know 
O  Not  applicable 

2.  Choose  the  term  below  which  best  describes  your  organization: 

O  Fleet  Command  (CINCLANT,CINCPAC,COMTHIRDFLT, . . ) 

O  Type  Command  (COMNAVAIRPAC,COMNAVAJRLANT,..) 

O  Systems  Command  (COMNAVAIRSYSCOM,SPAWAR,...) 

O  COMNAVRESFOR 

O  COMNAVAIRESFOR 
O  Naval  Safety  Center 
O  Marine  Corps  Activity 
O  Navy  Inventory  Control  Point 
O  Depot 
O  AIMD 
O  WING 
O  Squadron 

O  Civilian  Firm  under  contract  to  the  Navy 
O  Other 

3.  How  long  has  it  been  since  you  last  accessed  LMDSS? 

O  Less  than  6  months 

O  6  -12  months 
0  1-2  years 
O  2- 3 years 
0  3-4  years 
0  4-5  years 
O  More  than  5  years 
O  Don't  remember 
O  Not  applicable 

4.  When  were  you  first  introduced  to  LMDSS? 

O  Fewer  than  6  months  ago 

O  6 -12  months  ago 
0  1-2  years  ago 
O  2-3yearsago 
0  3-4  years  ago 
0  4-5  years  ago 
O  More  than  5  years  ago 


O  Don't  remember 
O  Not  applicable 

5.  When  the  LMDSS  Web  site  comes  back  online,  what  will  be  your  primary  means 
of  accessing  it? 

O  28K  modem 
O  56K  modem 
O  ISDN  connection 
O  T-l  connection 
O  Via  the  network 
O  Don't  know 
O  Other 

6.  What  kind  of  problems  did  you  have  accessing  LMDSS?  (Check  all  that  apply) 

□  I  have  not  had  any  problems. 

□  Firewall  blocking  access 

□  Proxy  difficulties 

□  Downloading  matrix  tables 

□  Unable  to  access  the  IQ  tool 

□  Computer  hardware  inadequate 

□  Internet  browser  issues 

□  Other 

□  Don't  know,  just  couldn't  access  the  LMDSS  web  site 

7.  When  LMDSS  is  back  online,  users  will  be  required  to  use  Netscape 
Communicator  4.5  with  128  bit  encryption  (available  for  free  download  at 
www.netscape.com).  Do  you  anticipate  any  problems  downloading  and  installing 
this  browser? 

O  Yes 
O  No 

O  Don't  know 
O  Not  applicable 

8.  What  kind  of  software  problems  did  you  experience  while  using  LMDSS?  (Write 
NA  or  "No  problems",  if  appropriate) 


9.  What  kind  of  information  do  you  expect  to  retrieve  from  LMDSS  when  it 
becomes  available?  (Write  "Don't  Know"  or  NA,  if  appropriate) 
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10.  What  information  do  you  require,  but  are  unable  to  access  via  LMDSS?  (Write 
"Don't  Know"  or  NA,  if  appropriate) 


1 1 .  How  difficult  was  it  to  build  matrix  tables  within  LMDSS? 

O  1  Impossible 
O  2 
O  3 
O  4 

O  5  Easy 
O  Not  applicable 

1 2.  Which  LMDSS  modules  did  you  access  the  most  (check  all  that  apply)? 

□  Management  Analysis 

□  Candidate  Identification 

□  Trend  Analysis 

□  Cost  Analysis 

□  Detailed  Analysis 

□  Supply  Analysis 

□  Engine  Analysis 

□  Reference  Information 

□  Application  Management  Tools 

□  Change  Requests 

□  Feature  Synopsis 

□  Other 

□  Don't  know 

□  Not  applicable 

1 3.  Have  you  ever  used  the  IQ  tool  to  build  any  custom  queries? 

O  Yes 

O  No 

O  Don't  know 
O  Not  applicable 

14.  Do  you  think  the  IQ  tool  was  user  friendly? 

O  Yes 

O  No 

O  Don't  know 
O  Not  applicable 


1 5.  Do  you  think  LMDSS  was  user  friendly? 
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O  Yes 
O  No 

O  Don't  know 

16.  How  important  do  you  consider  the  addition  of  a  structured,  ad-hoc  query  tool  to 
LMDSS? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don't  know 
O  Not  applicable 

1 7.  If  you  know  of  other  NALDA  applications  which  contain  query  formats  you 
would  like  to  see  implemented  in  LMDSS  please  list  them  below?  (Write  'Don’t 
know"  or  NA,  if  appropriate) 


1 8.  Please  list  a  few  of  the  key  decisions  you  would  like  LMDSS  to  support  when  it 
comes  back  online.  (Write  "Don't  Know"  or  NA,  if  appropriate) 


1 9.  LMDSS  will  collect  much  of  its  raw  data  from  NALCOMIS.  If  there  is  any  data 
you  will  require  which  is  not  captured  by  NALCOMIS,  please  describe  it  below? 
(Write  "Don't  know"  or  NA,  if  appropriate) 


20.  How  important  do  you  consider  the  development  of  a  modeling/simulation 
capability  in  future  versions  of  LMDSS? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don’t  know 
O  Not  applicable 
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21 .  Can  you  provide  an  example  of  how  a  modeling/simulation  capability  to  conduct 
sensitivity  (what  if)  analysis  will  be  useful?  (Write  "NO"  or  NA,  if  appropriate) 


22.  How  important  is  it  for  you  to  know  the  details  concerning  how  LMDSS  reports 
are  derived  before  you  feel  you  can  trust  the  information  they  provide? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don’t  know 
O  Not  applicable 

23.  How  important  is  a  detailed  description  or  definition  (data  dictionary)  of  all 
LMDSS  data? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don't  know 
O  Not  applicable 

24.  How  important  do  you  consider  the  development  of  a  graphics  capability  in 
future  versions  of  LMDSS? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don't  know 
O  Not  applicable 

25.  Have  you  found  the  usage  of  definitions  and  metrics  by  LMDSS  to  be  in 
compliance  with  established  3M  definitions? 

O  Yes 
O  No 

O  Don't  know 
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26.  How  would  you  rate  the  quality  of  the  LMDSS  instruction  you  have  received 
from  NAVAJR  personnel? 

O  Did  not  receive  any  training 
O  Excellent 
O  Good 
O  Fair 
O  Poor 

O  Don't  remember 
O  Not  applicable 

27.  Did  the  training  include  Structured  Query  Language  training  using  the  IQ  tool? 
O  Yes 

O  No 

O  Don't  remember 
O  Not  applicable 

28.  Do  you  feel  you  will  need  LMDSS  refresher  training  when  LMDSS  comes  back 
online? 

O  Yes 
O  No 

O  Don't  know 

29.  How  important  is  "hands  on"  LMDSS  training  with  each  student  assigned  to  a 
computer  terminal? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don't  know 
O  Not  applicable 

30.  How  can  LMDSS  training  be  improved?  (Write  "Don't  Know"  or  NA,  if 
appropriate) 


31 .  Please  describe  any  examples  of  your  not  being  satisfied  with  the  quality  of  the 
data  you  have  obtained  through  LMDSS?  (Write  "None"  or  NA  if  appropriate) 
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32.  How  important  is  it  for  future  versions  of  LMDSS  to  be  able  to  cleanly  export 
data  to  applications  with  graphics  capabilities  such  as  Excel? 

O  Extremely  important 
O  Very  important 
O  Somewhat  important 
O  Not  very  important 
O  Not  at  all  important 
O  Don't  know 
O  Not  applicable 

33.  Has  anyone  from  NAVAIR  or  the  LMDSS  development  team  ever  contacted  you 
requesting  your  input/feedback  concerning  LMDSS? 

O  Yes 
O  No 

O  Don't  know 

34.  Have  you  ever  attended  a  NAVAIR  sponsored  LMDSS  users'  group  meeting? 

O  Yes 

O  No 

O  Don't  remember 
O  Not  Applicable 

35.  What  might  some  of  your  reasons  be  for  using  LMDSS  when  it  returns  to  the 
Web?  (Check  all  that  apply) 

□  Find  data  to  support  logistics  acquisition  decisions 

□  Reduce  life  cycle  support  costs  of  aviation  systems 

□  Conduct  trend  analysis 

□  Search  for  data  to  complete  periodic  reports  and  forms 

□  Identify  high  cost  drivers  to  conduct  cost  analysis 

□  Track  the  results  of  implemented  improvements 

□  Measure  the  impact  of  decisions  or  policies  on  platform  readiness 

□  Measure  the  effect  of  improvement  actions  on  support  costs 

□  Identify  system  degraders 

□  View  cost  data  and  flight  hour  history  of  aircraft  engines 

□  Drill  down  to  the  MAF  level  to  determine  cause  of  system  degraders 

□  Compare  the  performance  of  my  command  with  similar  commands 

□  Reliability  information 

□  Other  (Please  describe  in  question  #  43) 

36.  If  you  selected  "other"  in  the  previous  question,  please  describe  below: 
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37.  Referring  to  your  answers  in  question  #  35  and  #  36,  what  do  you  anticipate  your 
primary  reason  for  using  LMDSS  will  be? 


38.  In  what  ways  could  NAVAIR  make  analysts  more  aware  of  LMDSS?  (Write 
"Don't  Know"  or  NA  if  appropriate) 


39.  If  you  have  any  additional  thoughts  as  to  how  NAVAIR  could  improve  LMDSS 
please  write  them  below:  (Write  "Don't  Know"  or  NA  if  appropriate) 
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APPENDIX  E.  PHONE  INTERVIEW  QUESTIONS 


1.  What  activity/command  do  you  work  for? 

What  is  your  job  title?  Describe  your  background  with  IT  systems.  How  many 
years  of  experience  do  you  have  with  the  3M  system?  In  what  capacities  have  you 
worked  with  the  3M  system? 

2.  Are  you  an  active  LMDSS  user? 

No:  Why  don’t  you  use  LMDSS?  Why  do  you  still  have  an  active  LMDSS 
account?  What  would  you  want  LMDSS  to  do  to  motivate  you  to  use  it  regularly? 

Yes:  How  often  do  you  use  LMDSS? 

3.  How  do  you  access  LMDSS? 

How  do  you  connect  to  the  Internet?  (network,  modem,  ISDN,  T-l)  Any 
problems  connecting  to  LMDSS  via  the  WWW?  What  kind  of  computer  do  you  use  to 
access  LMDSS?  (CPU  type/speed,  RAM,  hard  drive)  Any  problems  with  firewalls?  Any 
software  problems  related  to  LMDSS?  What  kind  of  browser  do  you  use?  Any  problems 
downloading  matrices? 

4.  Have  you  received  LMDSS  training  from  NAVAIR? 

When  was  the  training?  Where  did  you  receive  the  training?  Did  the 
instructors  effectively  teach  the  material?  Did  the  training  include  SQL  training?  How 
can  LMDSS  training  be  improved?  Do  you  feel  you  need  refresher  training.  Did  the 
training  adequately  prepare  you  to  effectively  use  LMDSS?  Why  or  why  not? 

5.  Have  you  used  the  Help  Desk? 

No:  Why  not? 
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Yes:  What  is  your  opinion  of  the  performance  of  the  Help  Desk?  How  often 
do  you  call  the  Help  Desk?  Any  problems  reaching  the  Help  Desk?  Are  the  Help  Desk 
personnel  able  to  adequately  answer  your  questions?  What  are  your  most  common  Help 
Desk  requests? 

6.  How  useful  is  LMDSS? 

What  questions  are  you  usually  trying  to  answer  with  LMDSS?  Do  you 
consider  LMDSS  to  be  user  friendly?  What  information  does  LMDSS  not  provide?  Any 
problems  building  the  matrices?  Which  matrices  do  you  access  the  most?  Is  the 
information  easy  to  get  from  LMDSS  or  are  there  too  many  steps?  Does  the  flow  of  data 
seem  logical  and  easy  to  follow? 

7.  Have  you  used  the  IQ  tool  to  build  any  SQL  queries? 

Do  you  feel  comfortable  using  the  IQ  tool  to  build  SQL  queries?  Is  the  IQ  tool 
user  friendly?  Should  a  more  structured,  ad-hoc  query  tool  be  developed  that’s  easier  to 
use?  Do  any  other  NALDA  applications  contain  query  formats  you  would  like  to  see 
used  in  LMDSS? 

8.  What  are  the  key  decisions  you  would  like  LMDSS  to  support? 

Does  LMDSS  provide  the  information  you  need  to  make  these  decisions? 
What  data  do  you  need  which  is  not  captured  by  NALCOMIS? 

9.  How  would  a  modeling  capability  be  helpful  in  supporting  your 
research? 

Describe  modeling  as  models  or  simulations,  which  allow  the  user  to  test 
“what  if’  scenarios.  Discuss  whether  sensitivity  analysis  (changing  variables  to  see  how 
the  outcome  is  affected)  is  an  important  capability  for  LMDSS  users.  Probe  for  specific 
examples  or  stories  that  show  how  useful  modeling  might  be  for  LMDSS  users. 
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10.  How  would  a  graphics  capability  be  useful? 

Probe  for  specific  examples  or  stories.  Would  the  capability  to  export  to  an 
application  with  a  graphics  capability  such  as  Microsoft’s  Excel  be  useful? 

11.  What  is  the  best  way  NAVAIR  could  make  analysts  more  aware  of 
LMDSS? 

How  did  you  become  aware  of  the  LMDSS  and  its  capabilities? 

12.  How  would  you  improve  LMDSS  to  make  it  better? 

Probe  for  examples. 
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APPENDIX  F.  SH-60B  LOGISTICS  MODEL 


As  shown  in  the  review  of  the  literature,  models  provide  a  DSS  with  powerful 
analytical  capabilities.  Sensitivity  analysis  using  models  and  simulations  allows 
managers  to  change  problem  variables  to  measure  their  impact  on  the  final  outcome  of  a 
proposed  alternative  solution.  Although  LMDSS  does  not  contain  models  or  simulations, 
this  appendix  provides  an  example  of  how  modeling  can  assist  managers  in  the 
evaluation  of  alternatives  to  a  real  world,  unstructured  problem. 

The  SH-60B  LAMPS  MKIII  helicopter  community  has  been  struggling  to 
overcome  a  readiness  degrader  involving  a  shortage  of  Ready  for  Installation  (RFI)  main 
rotor  blade  spindles.  The  primary  cause  of  the  shortage  is  the  backlog  of  spindles 
awaiting  maintenance  at  the  H-60  Depot  in  Corpus  Christi,  Texas.  The  Depot  is  tasked 
to  repair  H-60  spindles  submitted  for  overhaul  by  both  the  Army  and  Navy.  Due  to 
shortages  in  manpower,  training,  and  funding  the  Depot  has  been  unable  to  keep  up  with 
the  demand  for  RFI  spindles.  As  a  result.  Navy  SH-60B  helicopters  often  remain  in  a 
None  Mission  Capable  (NMC)  status  for  one  to  three  months  awaiting  a  RFI  spindle 
from  the  supply  system. 

The  purpose  of  this  model  is  to  provide  NAVAIR  3.6.2  with  a  prototype  tool 
capable  of  sensitivity  analysis  and  "what  if'  projections.  The  global  time  variable  for  the 
model  is  in  hours.  The  time  frame  for  each  model  run  is  88,400  hours  or  approximately 
10  calendar  years.  Figures  18  and  19  are  the  two  output  graphs  provided  at  the  end  of 
each  model  run.  Figure  18  shows  the  result  of  a  120  hour  (5  day)  delay  at  the  Depot 
given  the  assumption  there  are  336  "flying"  spindles  assigned  to  84  helicopters  in  the 
Pacific  Fleet  (PACFLT)  and  20  spare  spindles  in  the  supply  system.  In  Figure  18  the  top 
blue  area  of  the  graph  shows  only  304  RFI  spindles  in  PACFLT  after  10  years  with  the 
black  line  showing  52  of  the  total  356  spindles  are  in  the  Depot  awaiting  maintenance. 
The  red  line  depicting  the  number  of  spindles  in  the  supply  system  is  hidden  within  the 
horizontal 
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graph  axis  showing  a  steady  value  of  zero  since  the  Depot  never  catches  up  enough  to 
provide  any  spare  spindles  to  the  supply  system. 

Figure  19  shows  the  impact  on  the  readiness  of  PACFLT's  SH-60Bs  due  solely  to 
a  nonavailability  of  main  rotor  spindles.  The  plot  shows  the  readiness  shortfall  (from  a 
maximum  of  100  percent)  caused  by  the  H-60  Depot's  delay  in  providing  RFI  spindles  to 
PACFLT's  84  SH-60Bs  flying  approximately  100  horns  a  month.  Over  a  ten  year  period, 
the  Depot  delay  eventually  causes  a  readiness  degradation  of  approximately  ten  percent. 

Figures  20  through  22  show  the  actual  model  in  an  object-oriented  format.  Extend 
modeling  software  was  used  to  create  the  SH-60B  Logistics  Model.  The  model  is  divided 
into  various  sections  simulating  helicopters  (each  containing  4  spindles)  flying  3.3  hours 
each  day  at  the  squadron,  spindles  being  checked  and  removed  at  5000  flight  hours, 
spindles  being  repaired  at  the  Depot,  and  RFI  spindles  being  passed  to  supply  for  transfer 
back  to  the  Fleet.  Attributes  such  as  flight  hours  remaining  to  overhaul  and  daily  flight 
hours  are  assigned  to  each  spindle  as  it  passes  through  the  system.  Attribute  generators 
are  assigned  constant  values  or  probability  distributions  with  means  and  standard 
deviations  to  approximate  model  variables.  The  Depot  delay,  for  example,  is  initiated 
from  a  random  input  generator  assigned  a  normal  distribution  with  a  mean  of  120  hours 
(five  days)  and  a  standard  deviation  of  twelve  hours. 

The  power  of  this  model  is  that  variables  such  as  the  length  of  the  Depot  delay, 
helicopter  flight  hours  flown  per  month,  the  time  period  of  the  model,  and  the  number  of 
surplus  spindles  in  the  supply  system  can  all  be  changed  prior  to  each  run  to  see  the 
impact  on  the  final  spindle  readiness  of  PACFLT's  helicopters.  With  a  few  clicks  of  a 
mouse  on  selected  objects  shown  in  Figures  20  through  22,  any  single  or  multiple 
variable  change  can  be  accomplished.  The  output  graphs  provide  a  clear  visual  display 
of  the  results  of  those  changes  to  help  strategic  managers  determine  where  to  best  apply 
funding  to  achieve  desired  results. 
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Figure  22  Depot  and  Squadron  Sections  of  the  SH-60B  Model 
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The  data  for  the  SH-60B  Logistics  Model  were  provided  by  the 

COMNAVAIRPAC's  LAMPS  MK  III  Wing.  The  following  assumptions  are  simulated 

within  this  model: 

•  The  LAMPS  MKIII  Wing  "owns"  84  SH-60B  helicopters.  This  model  only 
considers  these  84  aircraft  and  does  not  take  into  account  additional  helicopters 
owned  by  the  Naval  Reserve,  Army,  and  the  Atlantic  Fleet. 

•  A  main  rotor  spindle  must  be  removed  for  maintenance  every  5000  flight  hours 
and  sent  to  the  Depot  for  overhaul. 

•  Each  SH-60B  flies  an  average  of  100  flight  hours  a  month  and  has  four  spindles, 
one  for  each  rotor  blade. 

•  A  spindle  has  never  failed  in  flight.  Spindles  are  only  removed  at  the  mandatory 
5000  flight  hour  maintenance  interval.  Cannibalizations  or  swapping  spindles 
between  aircraft  to  optimize  readiness  was  not  simulated.  A  helicopter  is  "down" 
or  non-flyable  without  a  RFI  spindle  installed. 

•  The  average  tum-around  time  for  a  spindle  includes  three  days  transit  time  from 
squadron  to  Depot,  five  days  repair  time  at  the  Depot,  three  days  transit  time  back 
to  the  squadron,  and  three  days  to  reinstall  the  spindle. 

•  It  was  assumed  there  are  approximately  20  surplus  spindles  in  the  supply  system. 

•  There  are  usually  zero  available  RFI  spindles  in  the  supply  system  each  month. 

•  Once  a  RFI  spindle  is  received  by  the  squadron,  it  takes  3  days  to  install  the 
spindle  and  complete  the  post-maintenance  check  flight. 

•  To  date,  a  spindle  has  never  been  received  as  non-RFI  from  the  H-60  Depot. 
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